Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

support loading parquet files in addition to jsonl.gz #763

Closed
percyliang opened this issue Oct 12, 2024 · 1 comment · Fixed by #764 or #766
Closed

support loading parquet files in addition to jsonl.gz #763

percyliang opened this issue Oct 12, 2024 · 1 comment · Fixed by #764 or #766
Assignees
@nikil-ravi
Labels
enhancement good first issue Good for newcomers

Comments

@percyliang
Copy link

This will allow us to more easily train on datasets such as SlimPajama-6B, fineweb-edu, etc.
@nikil-ravi nikil-ravi mentioned this issue Oct 13, 2024
Merged
nikil-ravi added a commit that referenced this issue Oct 14, 2024
@nikil-ravi
Support Parquet files in ShardedDataSource (#764)
fc26c74 
This PR creates a `ParquetDataSource` class to support loading
`.parquet` files.
Closes #763
@nikil-ravi nikil-ravi reopened this Oct 15, 2024
@nikil-ravi
Copy link
Contributor

Re-opened to address David's comments in #764

@nikil-ravi nikil-ravi mentioned this issue Oct 15, 2024
Merged
@dlwh dlwh closed this as completed in #766 Oct 15, 2024
@dlwh dlwh closed this as completed in 877ca7e Oct 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nikil-ravi nikil-ravi

Labels
enhancement good first issue Good for newcomers
Projects
None yet
Milestone
No milestone
2 participants
@percyliang @nikil-ravi