configs in the paper

[KimJaehee0725]

Hello! Thank you for this wonderful work.
It will help my recent work for training private llm :)

I have a question regarding the configurations among in the recipe page
I can find various recipes with the language(eng/chinese), and versions.

Which is the exact configurations for the reported model in the original paper, especially the 'MAGPIE-Pro-300K-Filtered' in the Table 1?

Thanks!

[zhangchen-xu]

It's this: https://huggingface.co/Magpie-Align/Llama-3-8B-Magpie-Pro-SFT-300K-v0.1.

You can go to the end of the page to see the detailed Axolotl configs. Note that since Axolotl has some major updates recently, you may need to slightly modify the configurations in datasets by changing:

datasets:
  - path: Magpie-Align/Magpie-Pro-300K-Filtered
    type: sharegpt
    conversation: llama3

to something like this:

chat_template: llama3
datasets:
  - path: Magpie-Align/Magpie-Pro-300K-Filtered
    type: chat_template
    field_messages: conversations
    # The key in the message turn that contains the role. Default is "role".
    message_field_role: from
    # The key in the message turn that contains the content. Default is "content".
    message_field_content: value
    # Optional[Dict[str, List]]. Roles mapping for the messages.
    roles:
      user: ["human", "user"]
      assistant: ["gpt", "assistant", "ai"]
      system: ["system"]

[KimJaehee0725]

thank you for your detailed explanation! I will follow the config you mentioned.

[KimJaehee0725]

Hello @zhangchen-xu
Thanks again to your reply.

Could you provide the config for evaluation with lm-evaluation-harness??

I want to reproduce the performance of Llama-3-8B on MMLU task.

[zhangchen-xu]

I am using Lighteval: https://github.com/huggingface/lighteval with its default settings.