Skip to content

Latest commit

 

History

History
140 lines (111 loc) · 5.06 KB

README.md

File metadata and controls

140 lines (111 loc) · 5.06 KB
Raw

Text classification using LLaMA

This repository provides a basic codebase for text classification using LLaMA.

What system do I use for development?

  • Device: Nvidia 1xV100 GPU
  • Device Memory: 34G
  • Host Memory: 252G

If you need other information about hardware, please open an issue.

How to use

Experimental setup

  1. Get the checkpoint from official LLaMA repository from here.
    1-1. I assume that the checkpoint would be located in the project root direction and the contents would be arranged as follow.

    checkpoints
├── llama
│   ├── 7B
│   │   ├── checklist.chk
│   │   ├── consolidated.00.pth
│   │   └── params.json
│   └── tokenizer.model

  2. Prepare your python environment. I recommend using anaconda to segregate your local machine CUDA version.

    conda create -y -n llama-classification python=3.8
conda activate llama-classification
conda install cudatoolkit=11.7 -y -c nvidia
conda list cudatoolkit # to check what cuda version is installed (11.7)
pip install -r requirements.txt

Method: Direct

Direct is to compare the conditional probability p(y|x).

  1. Preprocess the data from huggingface datasets using the following scripts. From now on, we use the ag_news dataset.

    python run_preprocess_direct_ag_news.py
python run_preprocess_direct_ag_news.py --sample=False --data_path=real/inputs_direct_ag_news.json # Use it for full evaluation

  2. Inference to compute the conditional probability using LLaMA and predict class.

    torchrun --nproc_per_node 1 run_evaluate_direct_llama.py \
    --data_path samples/inputs_direct_ag_news.json \
    --output_path samples/outputs_direct_ag_news.json \
    --ckpt_dir checkpoints/llama/7B \
    --tokenizer_path checkpoints/llama/tokenizer.model

Calibration is to improve direct method with calibration method.

  1. Calibrate using the following command. 
    torchrun --nproc_per_node 1 run_evaluate_direct_calibrate_llama.py \
    --direct_input_path samples/inputs_direct_ag_news.json \
    --direct_output_path samples/outputs_direct_ag_news.json \
    --output_path samples/outputs_direct_calibrate_ag_news.json \
    --ckpt_dir checkpoints/llama/7B \
    --tokenizer_path checkpoints/llama/tokenizer.model

Method: Channel

Channel is to compare the conditional probability p(x|y).

  1. Preprocess the data from huggingface datasets using the following scripts. From now on, we use the ag_news dataset.

    python run_preprocess_channel_ag_news.py
python run_preprocess_channel_ag_news.py --sample=False --data_path=real/inputs_channel_ag_news.json # Use it for full evaluation

  2. Inference to compute the conditional probability using LLaMA and predict class.

    torchrun --nproc_per_node 1 run_evaluate_channel_llama.py \
    --data_path samples/inputs_channel_ag_news.json \
    --output_path samples/outputs_channel_ag_news.json \
    --ckpt_dir checkpoints/llama/7B \
    --tokenizer_path checkpoints/llama/tokenizer.model

Method: Pure generation

  1. To evaluate using generate mode, you can use the preprocessed direct version. 
    torchrun --nproc_per_node 1 run_evaluate_generate_llama.py \
    --data_path samples/inputs_direct_ag_news.json \
    --output_path samples/outputs_generate_ag_news.json \
    --ckpt_dir checkpoints/llama/7B \
    --tokenizer_path checkpoints/llama/tokenizer.model

Experiments

Dataset num_examples k method accuracy inference time
ag_news 7600 1 direct 0.7682 00:38:40
ag_news 7600 1 direct+calibrated 0.8567 00:38:40
ag_news 7600 1 channel 0.7825 00:38:37

Todo list

  • Implement channel method
  • Experimental report
    • Direct
    • Channel
    • Generation
  • Implement other calibration method
  • Support other dataset inside the huggingface datasets
  • Implement LLM.int8
  • Other evaluation metric to measure the different characteristic of foundation model (LLaMA)

Final remark

  • I am really appreciate for the LLaMA project team to publish a checkpoint and their efficient inference code. Much of work in this repository is done based on the official repository.
  • For the reader, don't hesitate to open issue or pull requests. You can give me..
    • Any issue about other feature requests
    • Any issue about the detailed implementation
    • Any discussion about the research direction

Citation

It would be welcome citing my work if you use my codebase for your research.

@software{Lee_Simple_Text_Classification_2023,
    author = {Lee, Seonghyeon},
    month = {3},
    title = {{Simple Text Classification Codebase using LLaMA}},
    url = {https://github.com/github/sh0416/llama-classification},
    version = {1.1.0},
    year = {2023}
}