Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cache realization states to avoid redundant checking of files #8135

Closed
wants to merge 3 commits into from

Conversation

yngve-sk
Copy link
Contributor

@yngve-sk yngve-sk commented Jun 12, 2024

(this is a draft that shows somewhat what it would look like if we added state monitoring, currently there is quite some room for refactoring/improvement in readability/more tests)

Example of the state:

self.to_dataframe()
                                                      value     timestamp
realization kind                key                                      
0           SNAKE_OIL_PARAM_BPR BPR_138_PERSISTENCE    True  1.718870e+09
                                BPR_555_PERSISTENCE    True  1.718870e+09
            SNAKE_OIL_PARAM_OP1 OP1_DIVERGENCE_SCALE   True  1.718870e+09
                                OP1_OCTAVES            True  1.718870e+09
                                OP1_OFFSET             True  1.718870e+09
                                OP1_PERSISTENCE        True  1.718870e+09
            SNAKE_OIL_PARAM_OP2 OP2_DIVERGENCE_SCALE   True  1.718870e+09
                                OP2_OCTAVES            True  1.718870e+09
                                OP2_OFFSET             True  1.718870e+09
                                OP2_PERSISTENCE        True  1.718870e+09
            TOP                 TOP                    True  1.718870e+09

TOP->TOP indicates that the parameter TOP was updated (no groups here), kind->key otherwise indicates RESPONSE TYPE -> RESPONSE KEY / PARAMETER GROUP -> PARAMETER KEY

Benchmarking (Testing on RGS on large plot case)

  • Migrations took about 10 minutes
  • Opening with combined datasets takes ~25 minutes when migrations are done
  • Computing the state takes 1m~1m30s per ensemble
  • Opening case with pre-combine ert takes ~30 sec
  • Opening case with this PR takes 2m26s, but 2m2s is spent re-computing the state, making it about the same if not slightly faster than pre-combine if we just store the state in a file.

Issue
Resolves #8125

@yngve-sk yngve-sk force-pushed the fix-slow-realmask-fn branch 7 times, most recently from cbbe0fc to 2b596bf Compare June 17, 2024 05:40
@yngve-sk yngve-sk marked this pull request as draft June 17, 2024 08:53
@yngve-sk yngve-sk force-pushed the fix-slow-realmask-fn branch 11 times, most recently from a7e4d37 to b77e582 Compare June 19, 2024 13:32
@yngve-sk yngve-sk marked this pull request as ready for review June 20, 2024 06:13
@yngve-sk yngve-sk force-pushed the fix-slow-realmask-fn branch 10 times, most recently from be88a6c to 14848be Compare June 21, 2024 05:46
@yngve-sk yngve-sk force-pushed the fix-slow-realmask-fn branch 11 times, most recently from 6ef893f to cef9004 Compare June 24, 2024 07:06
@yngve-sk yngve-sk requested a review from oyvindeide June 24, 2024 10:23
@yngve-sk yngve-sk force-pushed the fix-slow-realmask-fn branch from cef9004 to 102bf8a Compare June 24, 2024 11:30
@codecov-commenter
Copy link

codecov-commenter commented Jun 24, 2024

Codecov Report

Attention: Patch coverage is 96.10778% with 13 lines in your changes missing coverage. Please review.

Project coverage is 86.97%. Comparing base (7281a93) to head (15fe79f).

Files Patch % Lines
src/ert/storage/realization_state.py 93.64% 11 Missing ⚠️
src/ert/storage/local_ensemble.py 98.72% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #8135      +/-   ##
==========================================
+ Coverage   86.86%   86.97%   +0.10%     
==========================================
  Files         379      380       +1     
  Lines       23971    24293     +322     
  Branches      620      624       +4     
==========================================
+ Hits        20822    21128     +306     
- Misses       3068     3086      +18     
+ Partials       81       79       -2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@yngve-sk yngve-sk force-pushed the fix-slow-realmask-fn branch 2 times, most recently from a6bacbd to 2d34a27 Compare June 25, 2024 06:49
@yngve-sk yngve-sk force-pushed the fix-slow-realmask-fn branch from 2d34a27 to 15fe79f Compare June 25, 2024 09:56
@yngve-sk yngve-sk closed this Sep 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Checking if responses exist for realization is very slow
2 participants