dataset tokenization script improvements#106

Merged

jettjaniak merged 12 commits intomainfrom 105-fix-dataset-download-for-its-tokenization2

Apr 24, 2024

+182-116

Commits on Apr 20, 2024

Update tokenize_dataset.py to use load_dataset instead of load_validation_dataset

joshuawe
authored and
jettjaniak
committed
include tqdm

joshuawe
authored and
jettjaniak
committed
add split handling functionality

joshuawe
authored and
jettjaniak
committed
beauty fix

joshuawe
authored and
jettjaniak
committed
arg names & cosmetics
jettjaniak
committed
added scripts/demo_upload_in_chunks.py
jettjaniak
committed
Update tokenization script to upload to HF in chunks

Siwei Li
authored and
jettjaniak
committed
Uncomment the repo creation

Siwei Li
authored and
jettjaniak
committed
refactor, cosmetics
jettjaniak
committed
memory usage FIXMEs
jettjaniak
committed

Commits on Apr 21, 2024

fix memory usage issues
jettjaniak
committed

Commits on Apr 24, 2024

fix test_tokenization
jettjaniak
committed