Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement LocalEnsemble finalize function, which combines all realisations into one dataset. #5527

Closed
oyvindeide opened this issue Jun 2, 2023 · 3 comments

Comments

@oyvindeide
Copy link
Collaborator

oyvindeide commented Jun 2, 2023

This has potential performace improvements. We should do it explicitly in the run models after we have finished loading all data. Could potentially be done in a different thread (or process). Then load results manually should probably do this directly and not do the callbacks individually first.

@pinkwah pinkwah added this to SCOUT Jun 1, 2023
@oyvindeide oyvindeide converted this from a draft issue Jun 2, 2023
@pinkwah pinkwah self-assigned this Jun 23, 2023
@pinkwah pinkwah removed their assignment Aug 14, 2023
@pinkwah
Copy link
Contributor

pinkwah commented Aug 14, 2023

Gonna wait with implementing this until we have figured out the starting on an existing case issue.

@oyvindeide oyvindeide moved this from Todo to Backlog in SCOUT Sep 22, 2023
@oyvindeide
Copy link
Collaborator Author

First step should be testing the performance of loading data from a finalized dataset compared with loading from multiple. Hopefully using a finalized dataset should take advantage of lazy loading. We should also check on NFS. Summary data is the most important to check. They are currently saved under a summary group, and the least performant at the moment, as we load all datasets and just pick out a single key. If this is not significantly more performant with a finalized dataset we need to revisit the way we store summary data.

@oyvindeide oyvindeide moved this from Backlog to Todo in SCOUT Jan 30, 2024
@frode-aarstad frode-aarstad moved this from Todo to In Progress in SCOUT Jan 30, 2024
@frode-aarstad frode-aarstad self-assigned this Jan 30, 2024
@frode-aarstad frode-aarstad moved this from In Progress to Todo in SCOUT Jan 31, 2024
@frode-aarstad frode-aarstad removed their assignment Jan 31, 2024
@kvashchuka kvashchuka moved this from Todo to Backlog in SCOUT Feb 9, 2024
@eivindjahren
Copy link
Contributor

Resolved by #7563

@github-project-automation github-project-automation bot moved this from Backlog to Done in SCOUT Jun 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

4 participants