Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[8pt] Split CatFIM output library file to group by HUC2's #1331

Open
RobHanna-NOAA opened this issue Oct 28, 2024 · 1 comment
Open

[8pt] Split CatFIM output library file to group by HUC2's #1331

RobHanna-NOAA opened this issue Oct 28, 2024 · 1 comment
Assignees
Labels
CatFIM NWS Flood Categorical HAND FIM enhancement New feature or request High Priority

Comments

@RobHanna-NOAA
Copy link
Contributor

During testing of CatFIM 2.1, it was proven that the CatFIM output library files are too big. In fim 4.5.2.11, stage based library file was one giant csv file and was ~10 GiB and it was very difficult to load based on it's size. This also made it pretty hard and slow to copy around, load in GPKG and debug. Even in the program, merging the 1,000's of individual site gpkgs was very hard on the server as it was loading all of that into memory in order to create one file.

Currently in the gpkg output folder, every single site, magnitude and interval creates a new file.

With the bug from 4.4.0.0/4.5.2.11 for stage for intervals, we are now expecting more interval and an even larger outputs. Split the final output file in HUC2.

Plan:

  • create a new subfolder in the root dir of the catfim file called "outputs"
  • put all files including the csv;s into that folder. It will represent all files / folder that are final outputs and are already grouped to go to HydroVIS.
  • Using the gpkg output folder, load and roll up all files into their respective HUC2 gpkg.
@RobHanna-NOAA
Copy link
Contributor Author

Put on hold. Adding this feature create more problems than expected. It all means we would have problem with our new catfim comparison tool which an compare versions of outputs (ie.. 4.4.0.0 to 4.5.2.11, etc)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CatFIM NWS Flood Categorical HAND FIM enhancement New feature or request High Priority
Projects
None yet
Development

No branches or pull requests

1 participant