Add a cpu example of HuggingFace Transformers Agent (use vicuna-7b-v1.5) #9284
Conversation
yzheng124
changed the title
|
Hi @shane-huang , please take a look. Thanks! |
|
|
||
| # Generate results | ||
| print(f"Image path: {args.image_path}") | ||
| print(agent.run("Generate a caption for the 'image'", image=image)) |
There was a problem hiding this comment.
print the prompt as well so that readers will know this is for image caption generation.
| # BigDL-LLM Transformers INT4 Optimization for HuggingFace Transformers Agent | ||
| In this directory, you will find examples on how you could apply BigDL-LLM INT4 optimizations on HuggingFace Transformers Agent. For illustration purposes, we utilize the [lmsys/vicuna-7b-v1.5](https://huggingface.co/lmsys/vicuna-7b-v1.5) as the reference model. | ||
|
|
||
| ## 0. Requirements |
There was a problem hiding this comment.
Remove Requirements and Best Known Configurations, and put a reference to https://github.com/intel-analytics/BigDL/tree/main/python/llm/example/CPU/HF-Transformers-AutoModels/Model#recommended-requirements as other examples in HF-Transformers-AutoModels/Model
| source bigdl-nano-init | ||
| ``` | ||
|
|
||
| ## Example: Run an agent using `run()` API |
There was a problem hiding this comment.
remove the lines
Example: Run an agent using
run()API
In the example run_agent.py, we show a basic use case of usingrun()API to run an agent, which uses Vicuna model with BigDL-LLM INT4 optimizations.
python/llm/example/CPU/README.md
Outdated
| @@ -6,6 +6,7 @@ This folder contains examples of running BigDL-LLM on Intel CPU: | |||
| - [PyTorch-Models](PyTorch-Models): running any PyTorch model on BigDL-LLM (with "one-line code change") | |||
| - [Native-Models](Native-Models): converting & running LLM in `llama`/`chatglm`/`bloom`/`gptneox`/`starcoder` model family using native (cpp) implementation | |||
| - [LangChain](LangChain): running LangChain applications on BigDL-LLM | |||
| - [Application](Application): running Transformers applications on BigDl-LLM | |||
There was a problem hiding this comment.
change folder from CPU/Application to CPU/Applications
| a little girl holding a stuffed teddy bear | ||
| ``` | ||
|
|
||
| ### 3. Reference |
There was a problem hiding this comment.
remove section 3. No need to tell user how to download it. And using coco image is okay but using image source directly from flicker may cause issues.
….5) (#9284) * Add examples of HF Agent * Modify folder structure and add link of demo.jpg * Fixes of readme * Merge applications and Applications
Description
1. Why the change?
In this PR, we add an cpu examples of HuggingFace Transformers Agent. This example uses vicuna-7b-v1.5 and generates caption for an input image.
The
demo.jpgis from COCO2017 train set (000000264959.jpg).https://github.com/analytics-zoo/nano/issues/669