diff --git a/python/llm/example/CPU/Application/hf-agent/README.md b/python/llm/example/CPU/Applications/hf-agent/README.md similarity index 72% rename from python/llm/example/CPU/Application/hf-agent/README.md rename to python/llm/example/CPU/Applications/hf-agent/README.md index fb346147ea3d..d0712999aba6 100644 --- a/python/llm/example/CPU/Application/hf-agent/README.md +++ b/python/llm/example/CPU/Applications/hf-agent/README.md @@ -1,20 +1,12 @@ # BigDL-LLM Transformers INT4 Optimization for HuggingFace Transformers Agent -In this directory, you will find examples on how you could apply BigDL-LLM INT4 optimizations on HuggingFace Transformers Agent. For illustration purposes, we utilize the [lmsys/vicuna-7b-v1.5](https://huggingface.co/lmsys/vicuna-7b-v1.5) as the reference model. +In this example, we apply low-bit optimizations to [HuggingFace Transformers Agents](https://huggingface.co/docs/transformers/transformers_agents) using BigDL-LLM, which allows LLMs to use tools such as image generation, image captioning, text summarization, etc. -## 0. Requirements -To run the examples, we recommend using Intel® Xeon® processors (server), or >= 12th Gen Intel® Core™ processor (client). +For illustration purposes, we utilize the [lmsys/vicuna-7b-v1.5](https://huggingface.co/lmsys/vicuna-7b-v1.5) as the reference model. We use [lmsys/vicuna-7b-v1.5](https://huggingface.co/lmsys/vicuna-7b-v1.5) to create an agent. Then we ask the agent to generate the caption for an image from coco dataset, i.e. [demo.jpg](https://cocodataset.org/#explore?id=264959) -For OS, BigDL-LLM supports Ubuntu 20.04 or later, CentOS 7 or later, and Windows 10/11. +## 0. Requirements +To run this example with BigDL-LLM, we have some recommended requirements for your machine, please refer to [here](https://github.com/intel-analytics/BigDL/tree/main/python/llm/example/CPU/HF-Transformers-AutoModels/Model#recommended-requirements) for more information. -## Best Known Configuration on Linux -For better performance, it is recommended to set environment variables on Linux with the help of BigDL-Nano: -```bash -pip install bigdl-nano -source bigdl-nano-init -``` -## Example: Run an agent using `run()` API -In the example [run_agent.py](./run_agent.py), we show a basic use case of using `run()` API to run an agent, which uses Vicuna model with BigDL-LLM INT4 optimizations. ### 1. Install We suggest using conda to manage environment: ```bash @@ -61,6 +53,8 @@ numactl -C 0-47 -m 0 python ./run_agent.py #### [lmsys/vicuna-7b-v1.5](https://huggingface.co/lmsys/vicuna-7b-v1.5) ```log Image path: demo.jpg +== Prompt == +Generate a caption for the 'image' ==Explanation from the agent== I will use the following tool: `image_captioner` to generate a caption for the image. diff --git a/python/llm/example/CPU/Application/hf-agent/demo.jpg b/python/llm/example/CPU/Applications/hf-agent/demo.jpg similarity index 100% rename from python/llm/example/CPU/Application/hf-agent/demo.jpg rename to python/llm/example/CPU/Applications/hf-agent/demo.jpg diff --git a/python/llm/example/CPU/Application/hf-agent/run_agent.py b/python/llm/example/CPU/Applications/hf-agent/run_agent.py similarity index 92% rename from python/llm/example/CPU/Application/hf-agent/run_agent.py rename to python/llm/example/CPU/Applications/hf-agent/run_agent.py index 807cf2dc1754..98fca0c5d4d5 100644 --- a/python/llm/example/CPU/Application/hf-agent/run_agent.py +++ b/python/llm/example/CPU/Applications/hf-agent/run_agent.py @@ -44,5 +44,8 @@ agent = LocalAgent(model, tokenizer) # Generate results + prompt = "Generate a caption for the 'image'" print(f"Image path: {args.image_path}") - print(agent.run("Generate a caption for the 'image'", image=image)) + print('==', 'Prompt', '==') + print(prompt) + print(agent.run(prompt, image=image)) diff --git a/python/llm/example/CPU/README.md b/python/llm/example/CPU/README.md index 4b4b3b87a3bd..b6be45cd2767 100644 --- a/python/llm/example/CPU/README.md +++ b/python/llm/example/CPU/README.md @@ -6,7 +6,7 @@ This folder contains examples of running BigDL-LLM on Intel CPU: - [PyTorch-Models](PyTorch-Models): running any PyTorch model on BigDL-LLM (with "one-line code change") - [Native-Models](Native-Models): converting & running LLM in `llama`/`chatglm`/`bloom`/`gptneox`/`starcoder` model family using native (cpp) implementation - [LangChain](LangChain): running LangChain applications on BigDL-LLM -- [Application](Application): running Transformers applications on BigDl-LLM +- [Applications](Applications): running Transformers applications on BigDl-LLM ## System Support **Hardware**: