how handle datasets that might not fit into memory

[kjappelbaum]

if i implement the splitting methods in the base class we assume (using sklearn's train/test split) that the datasets fit into memory.

However, this might not be the case for all datasets in the future.

One option would be that those future datasets override methods that might cause running out of memory. Other option might be to have different base classes and the default one would be a InMemoryDataset

[kjappelbaum]

we might also use Apache Arrow under the hood (this is what HuggingFace does)