Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

26 thp v2 arrow POC #30

Draft
wants to merge 4 commits into
base: 26-THP_v2
Choose a base branch
from
Draft

26 thp v2 arrow POC #30

wants to merge 4 commits into from

Conversation

chrisbc
Copy link
Member

@chrisbc chrisbc commented Apr 18, 2024

FYI @chrisdicaprio this branch has my workings on doing more data-wrangling using arrow/numpy/pandas. Notes:

  • I need your help to understand the composite/component branch data structures
  • I "think" I understand how to do all the math ops in numba, but leaving implementation until above is sorted
  • I expect this approach to have at least one (maybe two) orders better performance than the parent branch
  • multiple IMTs will be processed together, and same approach is doable for sites. But may need to manage memory
  • I've included hazard_v2_mini.toml and ..micro.toml configs for easier/faster comparisons between this (ARROW) and parent branch (OG) methods.
  • new command line option --method (-M) which defaults to OG, or use ARROW to try the newer approach.
  • my exploratory tests are in tests/version2/test_arrow_aggregation.py

@chrisbc chrisbc marked this pull request as draft April 18, 2024 01:56
for composite_branch in self.composite_branches:
log.debug(f"composite_branch wt: {composite_branch.weight:.3f} is {count} of {self.n_composite_branches}")
for branch in composite_branch.branches:
# if composite_branch.weight > 0:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for componet_branch in composite_branch.compoent_branches:
    weight_composite *= weight

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants