Implementing more CodeLMs

[yilunzhao]

Working on #30 for this PR

[niansong1996]

Good work! Some questions:

• Do you have to update the huggingface transformers library version?
• On what device/GPU did you test the mathqa setting?

[niansong1996]

So LLAMA does not have an eos token?

[yilunzhao]

It defines the eos token as empty string: https://huggingface.co/decapoda-research/llama-7b-hf/blob/main/tokenizer_config.json

[yilunzhao]

• Do you have to update the huggingface transformers library version?

Yes. There has not been an official release that supports LLAMA, so I installed the transformer library from source.

pip install git+https://github.com/huggingface/transformers

• On what device/GPU did you test the mathqa setting?

The experimental setting was recorded in this google doc. I will use this doc for updates.

[niansong1996]

To reiterate the action items discussed in the meeting:

• add the command to install the library so we can replicate the results

Also I was wondering if this PR is ready to merge? You also mentioned that there are some edge cases that haven't been handled?

[yilunzhao]

• add the command to install the library so we can replicate the results

I just updated the requirements.txt in the second commit.

Also I was wondering if this PR is ready to merge? You also mentioned that there are some edge cases that haven't been handled?

Yes. LLAMA, Alpaca, and santacoder should work fine using the config file: /home/lily/yl2465/code/NLP4Code/finetuning/training_configs/few_shot/mathqa-8_fixed_mathqa_shots_llama.yaml

[niansong1996]

@yilunzhao Can you resolve the conflicts and also merge from main to run the CI tests?

[niansong1996]

Great, merging this PR now

[niansong1996]

@yilunzhao Check my comments on #46

Since we can't reopen a merged PR, can you submit a new PR and point it to this PR instead?

Let me know if you have any questions. Sorry about the confusion.

[yilunzhao]

Hi @niansong1996, I have submit a new PR #48, could you please have a look at it?
This time, I add an instruction in README about how to install transformers library from source to run LLAMA-based models, rather than modifying the requirement.txt.