Skip to content

Commit

Permalink
Use ShardBasedBuilder for HuggingFace datasets
Browse files Browse the repository at this point in the history 
This is a more efficient way and simpler way to generate HuggingFace datasets. This can save a lot of time and space, especially for large datasets.

This works on Beam and non-Beam.

PiperOrigin-RevId: 682225015
  • Loading branch information
@tomvdw
tomvdw authored and The TensorFlow Datasets Authors committed Oct 4, 2024
1 parent 26812e9 commit e227519
Show file tree
Hide file tree
Showing 2 changed files with 104 additions and 217 deletions.
Loading

0 comments on commit e227519

Please sign in to comment.