notebooks/6.1-fine-tuning-walkthrough-hugging-face.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "source for this notebook: https://huggingface.co/blog/llama3#fine-tuning-with-%F0%9F%A4%97-trl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install --upgrade transformers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using Llama 3 with Transformers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following snippet shows how to use Llama-3-8b-instruct with transformers. It requires about 16 GB of RAM, which includes consumer GPUs such as 3090 or 4090."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import pipeline\n",
    "import torch\n",
    "\n",
    "model_id = \"meta-llama/Meta-Llama-3-8B-Instruct\"\n",
    "\n",
    "pipe = pipeline(\n",
    "    \"text-generation\",\n",
    "    model=model_id,\n",
    "    model_kwargs={\"torch_dtype\": torch.bfloat16},\n",
    "    device=\"cuda\",\n",
    ")\n",
    "\n",
    "messages = [\n",
    "    {\"role\": \"system\", \"content\": \"You are a pirate chatbot who always responds in pirate speak!\"},\n",
    "    {\"role\": \"user\", \"content\": \"Who are you?\"},\n",
    "]\n",
    "\n",
    "terminators = [\n",
    "    pipe.tokenizer.eos_token_id,\n",
    "    pipe.tokenizer.convert_tokens_to_ids(\"<|eot_id|>\")\n",
    "]\n",
    "\n",
    "outputs = pipe(\n",
    "    messages,\n",
    "    max_new_tokens=256,\n",
    "    eos_token_id=terminators,\n",
    "    do_sample=True,\n",
    "    temperature=0.6,\n",
    "    top_p=0.9,\n",
    ")\n",
    "assistant_response = outputs[0][\"generated_text\"][-1][\"content\"]\n",
    "print(assistant_response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. We loaded the model in bfloat16. This is the type used by the original checkpoint published by Meta, so it’s the recommended way to run to ensure the best precision or to conduct evaluations. For real world use, it’s also safe to use float16, which may be faster depending on your hardware.\n",
    "\n",
    "2. Assistant responses may end with the special token <|eot_id|>, but we must also stop generation if the regular EOS token is found. We can stop generation early by providing a list of terminators in the eos_token_id parameter.\n",
    "\n",
    "3. We used the default sampling parameters (temperature and top_p) taken from the original meta codebase. We haven’t had time to conduct extensive tests yet, feel free to explore!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also automatically quantize the model, loading it in 8-bit or even 4-bit mode. 4-bit loading takes about 7 GB of memory to run, making it compatible with a lot of consumer cards and all the GPUs in Google Colab. This is how you’d load the generation pipeline in 4-bit:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline = transformers.pipeline(\n",
    "    \"text-generation\",\n",
    "    model=model_id,\n",
    "    model_kwargs={\n",
    "        \"torch_dtype\": torch.float16,\n",
    "        \"quantization_config\": {\"load_in_4bit\": True},\n",
    "        \"low_cpu_mem_usage\": True,\n",
    "    },\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Inference Integrations\n",
    "In this section, we’ll go through different approaches to running inference of the Llama 3 models. Before using these models, make sure you have requested access to one of the models in the official Meta Llama 3 repositories.\n",
    "\n",
    "Integration with Inference Endpoints\n",
    "You can deploy Llama 3 on Hugging Face's Inference Endpoints, which uses Text Generation Inference as the backend. Text Generation Inference is a production-ready inference container developed by Hugging Face to enable easy deployment of large language models. It has features such as continuous batching, token streaming, tensor parallelism for fast inference on multiple GPUs, and production-ready logging and tracing.\n",
    "\n",
    "To deploy Llama 3, go to the model page and click on the Deploy -> Inference Endpoints widget. You can learn more about Deploying LLMs with Hugging Face Inference Endpoints in a previous blog post. Inference Endpoints supports Messages API through Text Generation Inference, which allows you to switch from another closed model to an open one by simply changing the URL."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from openai import OpenAI\n",
    "\n",
    "# initialize the client but point it to TGI\n",
    "client = OpenAI(\n",
    "    base_url=\"<ENDPOINT_URL>\" + \"/v1/\",  # replace with your endpoint url\n",
    "    api_key=\"<HF_API_TOKEN>\",  # replace with your token\n",
    ")\n",
    "chat_completion = client.chat.completions.create(\n",
    "    model=\"tgi\",\n",
    "    messages=[\n",
    "        {\"role\": \"user\", \"content\": \"Why is open-source software important?\"},\n",
    "    ],\n",
    "    stream=True,\n",
    "    max_tokens=500\n",
    ")\n",
    "\n",
    "# iterate and print stream\n",
    "for message in chat_completion:\n",
    "    print(message.choices[0].delta.content, end=\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fine Tuning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below is an example command to fine-tune Llama 3 on the No Robots dataset. We use 4-bit quantization, and QLoRA and TRL’s SFTTrainer will automatically format the dataset into chatml format. Let’s get started!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -U transformers trl accelerate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you just want to chat with the model in the terminal you can use the chat command of the TRL CLI (for more info see the docs):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!trl chat \\\n",
    "--model_name_or_path meta-llama/Meta-Llama-3-8B-Instruct \\\n",
    "--device cuda \\\n",
    "--eos_tokens \"<|end_of_text|>,<|eod_id|>\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also use TRL CLI to supervise fine-tuning (SFT) Llama 3 on your own, custom dataset. Use the trl sft command and pass your training arguments as CLI argument. Make sure you are logged in and have access the Llama 3 checkpoint. You can do this with huggingface-cli login."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!trl sft \\\n",
    "--model_name_or_path meta-llama/Meta-Llama-3-8B \\\n",
    "--dataset_name HuggingFaceH4/no_robots \\\n",
    "--learning_rate 0.0001 \\\n",
    "--per_device_train_batch_size 4 \\\n",
    "--max_seq_length 2048 \\\n",
    "--output_dir ./llama3-sft \\\n",
    "--use_peft \\\n",
    "--load_in_4bit \\\n",
    "--log_with wandb \\\n",
    "--gradient_checkpointing \\\n",
    "--logging_steps 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Meta Fine Tuning Recipes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Recipes PEFT LoRA\n",
    "source: https://llama.meta.com/docs/how-to-guides/fine-tuning/#:~:text=Recipes%20PEFT%20LoRA,gpu%20with%20FSDP.\n",
    "\n",
    "The llama-recipes repo has details on different fine-tuning (FT) alternatives supported by the provided sample scripts. In particular, it highlights the use of PEFT as the preferred FT method, as it reduces the hardware requirements and prevents catastrophic forgetting. For specific cases, full parameter FT can still be valid, and different strategies can be used to still prevent modifying the model too much. Additionally, FT can be done in single gpu or multi-gpu with FSDP.\n",
    "In order to run the recipes, follow the steps below:\n",
    "\n",
    "1. Create a conda environment with pytorch and additional dependencies\n",
    "2. Install the recipes as described here:\n",
    "3. Download the desired model from hf, either using git-lfs or using the llama download script.\n",
    "4. With everything configured, run the following command:\n",
    "\n",
    "```python -m llama_recipes.finetuning --use_peft --peft_method lora --quantization --model_name ../llama/models_hf/7B --output_dir ../llama/models_ft/7B-peft --batch_size_training 2 --gradient_accumulation_steps 2```\n",
    "[Torchtune option.](https://llama.meta.com/docs/how-to-guides/fine-tuning/#:~:text=2%20--gradient_accumulation_steps%202-,torchtune)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "oreilly-llama3",
   "language": "python",
   "name": "oreilly-llama3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}