Add OnDisk dataset, loader, preprocessor with Human3.6M example #132
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…o run.py and config file to allow dataset to be run OnDisk. Need to implement random split; previously only fixed was implemented
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Create classes so that datasets can be loaded OnDisk, using pointers, rather than InMemory. This requires an OnDisk dataset class (example is Human3.6M dataset, a standard benchmarking dataset for human motion prediction), OnDisk preprocessor and OnDisk dataloader. This also required additions to some config files and the general run.py file.
Still todo: Implement in the transductive setting and with k-fold splits
Issue
Some datasets are much too big to load into memory, and things crash. It is important to allow loading of things in a database on disk, as you need them, rather than having everything in memory at once. An example is Human3.6M dataset. On my gpu, it would crash things when trying to load and train with the entire thing in memory.