Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

translation evalution error #154

Open
laozhanghahaha opened this issue Apr 24, 2023 · 0 comments
Open

translation evalution error #154

laozhanghahaha opened this issue Apr 24, 2023 · 0 comments

Comments

@laozhanghahaha
Copy link

Hey I'm trying to evaluate bloom1b7 on translation task with this command
python main.py --model_api_name hf-causal --model_args pretrained=bigscience/bloom-1b7 --task_name flores_101_mt_fewshot_en2bn --device cuda:1

But I got this error

Traceback (most recent call last):
  File "/mnt/cephfs2/projects/dialogue/LLM/lm-evaluation-harness-ml/main.py", line 215, in <module>
    main()
  File "/mnt/cephfs2/projects/dialogue/LLM/lm-evaluation-harness-ml/main.py", line 197, in main
    results = evaluator.cli_evaluate(**evaluate_args)
  File "/mnt/cephfs2/projects/dialogue/LLM/lm-evaluation-harness-ml/lm_eval/evaluator.py", line 77, in cli_evaluate
    tasks = lm_eval.tasks.get_task_list_from_args_string(
  File "/mnt/cephfs2/projects/dialogue/LLM/lm-evaluation-harness-ml/lm_eval/tasks/__init__.py", line 327, in get_task_list_from_args_string
    return get_task_list(task_name, template_names, **kwargs)
  File "/mnt/cephfs2/projects/dialogue/LLM/lm-evaluation-harness-ml/lm_eval/tasks/__init__.py", line 281, in get_task_list
    assert template_names, "Must specify at least one template name"
AssertionError: Must specify at least one template name

So to get a proper template name I try to use this function lm_eval.list_templates()provided in READ.ME

>>> import lm_eval
>>> lm_eval.list_templates("flores_101_mt_fewshot_en2bn")
WARNING:root:Tried instantiating `DatasetTemplates` for gsarti/flores_101/all, but no prompts found. Please ignore this warning if you are creating new prompts for this dataset.
[]

It returns a empty list. Where could I get a template for flores_101_mt_fewshot_en2bn task

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant