New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

请问在处理微调数据集时为何要限制文本长度? #73

Open

jzzzf opened this issue May 12, 2024 · 1 comment

jzzzf commented May 12, 2024

if len(q) < 10 or len(a) < 5:
continue
if len(q) > 256 or len(a) > 256:
continue
请教下限制q,a长度的原因

buhe commented May 25, 2024

估计是太短效果不好，预训练也有类似判断

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment