Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adapt monarch to officially handle runs stretching across multiple files #20

Open
nsoblath opened this issue Apr 4, 2017 · 0 comments
Labels

Comments

@nsoblath
Copy link
Member

nsoblath commented Apr 4, 2017

The egg format was designed according to the "file == run" model. Runs can be artificially stretched across multiple files just by having the user of the monarch library take care of the details of what it means to have a multi-file run.

However, if multi-file runs are now the desired mode of operation, the egg3 format should be modified to handle that. Here's a no-doubt incomplete list of things that need to be changed:

  • Dealing with the header should be simplified, and assumptions can be made about the structure of the files that follow the first one (with appropriate checking, though we do want things to be fast).
  • Acquisitions need to be able to stretch across files. The concept of an acquisition needs to be separated from the HDF5 dataset that stores records.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant