New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

国内网/服务器网不能自动从huggingface datasets上下载文件的解决方案 #6

Open

ShengdingHu opened this issue Jul 11, 2022 · 1 comment

Labels

notes

Collaborator

ShengdingHu commented Jul 11, 2022

No description provided.

Collaborator Author

ShengdingHu commented Jul 11, 2022 •

edited

Loading

可以借助一个可以连huggingface datasets，并且可以将文件scp到目标服务器上的机器，例如本地机器, 运行python 代码

from datasets import load_dataset
mydataset = load_dataset("glue", "mrpc")
mydataset.save_to_disk("YOURPATH/glue.mrpc") # 不一定叫glue.mrpc 取个名就行

在终端中

scp -r  YOURPATH/glue.mprc  USERNAME@IP:THE_ABSOLUTE_PATH_TO_SAVE_YOUR_DATASET

之后在服务器中, 运行python代码

from datasets import load_from_disk
mydataset = load_from_disk(THE_ABSOLUTE_PATH_TO_SAVE_YOUR_DATASET)

ShengdingHu added the notes label

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment