Add examples for HF datasets #1371
Conversation
facebook-github-bot
added
the
CLA Signed
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/data/1371
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit d906a44 with merge base aebad0c ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
examples/nodes/hf_datsets.py
Outdated
| @@ -0,0 +1,82 @@ | |||
| from datasets import Dataset # make sure that you have huggingface datasets installed | |||
|
|
|||
| data = [[1, 2], [3, 4], [5, 6], [7, 8], [9, 10]] | |||
There was a problem hiding this comment.
Along with an simple example like this, I wonder if it would add value to include an actual HF Dataset - maybe a text one?
There was a problem hiding this comment.
Yeah, that's my goal as well.
I am facing some fwdproxy issues with loading the datasets from huggingface.
I am trying to resolve that, after that I can include examples with more complex datasets.
examples/nodes/hf_datsets.py
Outdated
|
|
||
| source_node = IterableWrapper(ds) | ||
|
|
||
| print("==" * 20) |
There was a problem hiding this comment.
One thing I had in mind with /examples, we don't expect users to really run the code, but mostly use it as a reference implementation to port over their workflows. I wonder if we can write this file such that users can see all the relevant nodes linked together clearly?
There was a problem hiding this comment.
Yes, sure.
I will port these examples to a Jupyter notebook with print outputs in the file so it is easier to get the outputs.
Will also add some interleaved examples where we use multiple node functionalities together.
Adding examples for HF datasets with nodes.
Fixes #1352