From 8d99c98cff6cad6060b665cf89e374b1ba267e8b Mon Sep 17 00:00:00 2001 From: jinbridge <2635480475@qq.com> Date: Thu, 13 Jun 2024 15:20:52 +0800 Subject: [PATCH] fix section --- .../Model/chatglm2/README.md | 20 ++++++++--------- .../Model/chatglm3/README.md | 20 ++++++++--------- .../Model/glm4/README.md | 22 +++++++++---------- .../Model/chatglm2/README.md | 22 +++++++++---------- .../Model/chatglm3/README.md | 22 +++++++++---------- .../Model/glm4/README.md | 22 +++++++++---------- .../PyTorch-Models/Model/chatglm2/README.md | 20 ++++++++--------- .../PyTorch-Models/Model/chatglm3/README.md | 20 ++++++++--------- 8 files changed, 84 insertions(+), 84 deletions(-) diff --git a/python/llm/example/CPU/HF-Transformers-AutoModels/Model/chatglm2/README.md b/python/llm/example/CPU/HF-Transformers-AutoModels/Model/chatglm2/README.md index 8efb927ba52..bc9a5bf6d9d 100644 --- a/python/llm/example/CPU/HF-Transformers-AutoModels/Model/chatglm2/README.md +++ b/python/llm/example/CPU/HF-Transformers-AutoModels/Model/chatglm2/README.md @@ -2,10 +2,10 @@ In this directory, you will find examples on how you could apply IPEX-LLM INT4 optimizations on ChatGLM2 models. For illustration purposes, we utilize the [THUDM/chatglm2-6b](https://huggingface.co/THUDM/chatglm2-6b) as a reference ChatGLM2 model. -### 0. Requirements +## 0. Requirements To run these examples with IPEX-LLM, we have some recommended requirements for your machine, please refer to [here](../README.md#recommended-requirements) for more information. -### 1. Install +## 1. Install We suggest using conda to manage environment: On Linux: @@ -27,9 +27,9 @@ conda activate llm pip install --pre --upgrade ipex-llm[all] ``` -### 2. Run +## 2. Run -#### Example 1: Predict Tokens using `generate()` API +### Example 1: Predict Tokens using `generate()` API In the example [generate.py](./generate.py), we show a basic use case for a ChatGLM2 model to predict the next N tokens using `generate()` API, with IPEX-LLM INT4 optimizations. ``` @@ -45,13 +45,13 @@ Arguments info: > > Please select the appropriate size of the ChatGLM2 model based on the capabilities of your machine. -##### 2.1 Client +#### 2.1 Client On client Windows machine, it is recommended to run directly with full utilization of all cores: ```cmd python ./generate.py ``` -##### 2.2 Server +#### 2.2 Server For optimal performance on server, it is recommended to set several environment variables (refer to [here](../README.md#best-known-configuration-on-linux) for more information), and run the example with all the physical cores of a single socket. E.g. on Linux, @@ -64,7 +64,7 @@ export OMP_NUM_THREADS=48 numactl -C 0-47 -m 0 python ./generate.py ``` -##### 2.3 Sample Output +#### 2.3 Sample Output ##### [THUDM/chatglm2-6b](https://huggingface.co/THUDM/chatglm2-6b) ```log Inference time: xxxx s @@ -90,7 +90,7 @@ Inference time: xxxx s 答： Artificial Intelligence (AI) refers to the ability of a computer or machine to perform tasks that typically require human-like intelligence, such as understanding language, recognizing patterns ``` -#### Example 2: Stream Chat using `stream_chat()` API +### Example 2: Stream Chat using `stream_chat()` API In the example [streamchat.py](./streamchat.py), we show a basic use case for a ChatGLM2 model to stream chat, with IPEX-LLM INT4 optimizations. **Stream Chat using `stream_chat()` API**: @@ -112,14 +112,14 @@ Arguments info: > > Please select the appropriate size of the ChatGLM2 model based on the capabilities of your machine. -##### 2.1 Client +#### 2.1 Client On client Windows machine, it is recommended to run directly with full utilization of all cores: ```cmd $env:PYTHONUNBUFFERED=1 # ensure stdout and stderr streams are sent straight to terminal without being first buffered python ./streamchat.py ``` -##### 2.2 Server +#### 2.2 Server For optimal performance on server, it is recommended to set several environment variables (refer to [here](../README.md#best-known-configuration-on-linux) for more information), and run the example with all the physical cores of a single socket. E.g. on Linux, diff --git a/python/llm/example/CPU/HF-Transformers-AutoModels/Model/chatglm3/README.md b/python/llm/example/CPU/HF-Transformers-AutoModels/Model/chatglm3/README.md index 99dc26c0491..5a14aec47d7 100644 --- a/python/llm/example/CPU/HF-Transformers-AutoModels/Model/chatglm3/README.md +++ b/python/llm/example/CPU/HF-Transformers-AutoModels/Model/chatglm3/README.md @@ -2,10 +2,10 @@ In this directory, you will find examples on how you could apply IPEX-LLM INT4 optimizations on ChatGLM3 models. For illustration purposes, we utilize the [THUDM/chatglm3-6b](https://huggingface.co/THUDM/chatglm3-6b) as a reference ChatGLM3 model. -### 0. Requirements +## 0. Requirements To run these examples with IPEX-LLM, we have some recommended requirements for your machine, please refer to [here](../README.md#recommended-requirements) for more information. -### 1. Install +## 1. Install We suggest using conda to manage environment: On Linux: @@ -27,9 +27,9 @@ conda activate llm pip install --pre --upgrade ipex-llm[all] ``` -### 2. Run +## 2. Run -#### Example 1: Predict Tokens using `generate()` API +### Example 1: Predict Tokens using `generate()` API In the example [generate.py](./generate.py), we show a basic use case for a ChatGLM3 model to predict the next N tokens using `generate()` API, with IPEX-LLM INT4 optimizations. ``` @@ -45,13 +45,13 @@ Arguments info: > > Please select the appropriate size of the ChatGLM3 model based on the capabilities of your machine. -##### 2.1 Client +#### 2.1 Client On client Windows machine, it is recommended to run directly with full utilization of all cores: ```cmd python ./generate.py ``` -##### 2.2 Server +#### 2.2 Server For optimal performance on server, it is recommended to set several environment variables (refer to [here](../README.md#best-known-configuration-on-linux) for more information), and run the example with all the physical cores of a single socket. E.g. on Linux, @@ -64,7 +64,7 @@ export OMP_NUM_THREADS=48 numactl -C 0-47 -m 0 python ./generate.py ``` -##### 2.3 Sample Output +#### 2.3 Sample Output ##### [THUDM/chatglm3-6b](https://huggingface.co/THUDM/chatglm3-6b) ```log Inference time: xxxx s @@ -91,7 +91,7 @@ What is AI? AI stands for Artificial Intelligence. It refers to the development of computer systems that can perform tasks that would normally require human intelligence, such as recognizing speech or making ``` -#### Example 2: Stream Chat using `stream_chat()` API +### Example 2: Stream Chat using `stream_chat()` API In the example [streamchat.py](./streamchat.py), we show a basic use case for a ChatGLM3 model to stream chat, with IPEX-LLM INT4 optimizations. **Stream Chat using `stream_chat()` API**: @@ -113,14 +113,14 @@ Arguments info: > > Please select the appropriate size of the ChatGLM3 model based on the capabilities of your machine. -##### 2.1 Client +#### 2.1 Client On client Windows machine, it is recommended to run directly with full utilization of all cores: ```cmd $env:PYTHONUNBUFFERED=1 # ensure stdout and stderr streams are sent straight to terminal without being first buffered python ./streamchat.py ``` -##### 2.2 Server +#### 2.2 Server For optimal performance on server, it is recommended to set several environment variables (refer to [here](../README.md#best-known-configuration-on-linux) for more information), and run the example with all the physical cores of a single socket. E.g. on Linux, diff --git a/python/llm/example/CPU/HF-Transformers-AutoModels/Model/glm4/README.md b/python/llm/example/CPU/HF-Transformers-AutoModels/Model/glm4/README.md index f62265f3d8a..db2a0c524bb 100644 --- a/python/llm/example/CPU/HF-Transformers-AutoModels/Model/glm4/README.md +++ b/python/llm/example/CPU/HF-Transformers-AutoModels/Model/glm4/README.md @@ -2,10 +2,10 @@ In this directory, you will find examples on how you could apply IPEX-LLM INT4 optimizations on GLM-4 models. For illustration purposes, we utilize the [THUDM/glm-4-9b-chat](https://huggingface.co/THUDM/glm-4-9b-chat) as a reference GLM-4 model. -### 0. Requirements +## 0. Requirements To run these examples with IPEX-LLM, we have some recommended requirements for your machine, please refer to [here](../README.md#recommended-requirements) for more information. -### 1. Install +## 1. Install We suggest using conda to manage environment: On Linux: @@ -32,9 +32,9 @@ pip install --pre --upgrade ipex-llm[all] pip install tiktoken ``` -### 2. Run +## 2. Run -#### Example 1: Predict Tokens using `generate()` API +### Example 1: Predict Tokens using `generate()` API In the example [generate.py](./generate.py), we show a basic use case for a GLM-4 model to predict the next N tokens using `generate()` API, with IPEX-LLM INT4 optimizations. ``` @@ -50,13 +50,13 @@ Arguments info: > > Please select the appropriate size of the GLM-4 model based on the capabilities of your machine. -##### 2.1 Client +#### 2.1 Client On client Windows machine, it is recommended to run directly with full utilization of all cores: ```cmd python ./generate.py ``` -##### 2.2 Server +#### 2.2 Server For optimal performance on server, it is recommended to set several environment variables (refer to [here](../README.md#best-known-configuration-on-linux) for more information), and run the example with all the physical cores of a single socket. E.g. on Linux, @@ -69,8 +69,8 @@ export OMP_NUM_THREADS=48 numactl -C 0-47 -m 0 python ./generate.py ``` -##### 2.3 Sample Output -###### [THUDM/glm-4-9b-chat](https://huggingface.co/THUDM/glm-4-9b-chat) +#### 2.3 Sample Output +##### [THUDM/glm-4-9b-chat](https://huggingface.co/THUDM/glm-4-9b-chat) ```log Inference time: xxxx s -------------------- Prompt -------------------- @@ -97,7 +97,7 @@ What is AI? Artificial Intelligence (AI) refers to the simulation of human intelligence in machines that are programmed to think like humans and mimic their actions. The term "art ``` -#### Example 2: Stream Chat using `stream_chat()` API +### Example 2: Stream Chat using `stream_chat()` API In the example [streamchat.py](./streamchat.py), we show a basic use case for a GLM-4 model to stream chat, with IPEX-LLM INT4 optimizations. **Stream Chat using `stream_chat()` API**: @@ -119,14 +119,14 @@ Arguments info: > > Please select the appropriate size of the GLM-4 model based on the capabilities of your machine. -##### 2.1 Client +#### 2.1 Client On client Windows machine, it is recommended to run directly with full utilization of all cores: ```cmd $env:PYTHONUNBUFFERED=1 # ensure stdout and stderr streams are sent straight to terminal without being first buffered python ./streamchat.py ``` -##### 2.2 Server +#### 2.2 Server For optimal performance on server, it is recommended to set several environment variables (refer to [here](../README.md#best-known-configuration-on-linux) for more information), and run the example with all the physical cores of a single socket. E.g. on Linux, diff --git a/python/llm/example/GPU/HF-Transformers-AutoModels/Model/chatglm2/README.md b/python/llm/example/GPU/HF-Transformers-AutoModels/Model/chatglm2/README.md index 71bde904881..9f6a268a30e 100644 --- a/python/llm/example/GPU/HF-Transformers-AutoModels/Model/chatglm2/README.md +++ b/python/llm/example/GPU/HF-Transformers-AutoModels/Model/chatglm2/README.md @@ -2,11 +2,11 @@ In this directory, you will find examples on how you could apply IPEX-LLM INT4 optimizations on ChatGLM2 models on [Intel GPUs](../../../README.md). For illustration purposes, we utilize the [THUDM/chatglm2-6b](https://huggingface.co/THUDM/chatglm2-6b) as a reference ChatGLM2 model. -### 0. Requirements +## 0. Requirements To run these examples with IPEX-LLM on Intel GPUs, we have some recommended requirements for your machine, please refer to [here](../../../README.md#requirements) for more information. -### 1. Install -#### 1.1 Installation on Linux +## 1. Install +### 1.1 Installation on Linux We suggest using conda to manage environment: ```bash conda create -n llm python=3.11 @@ -15,7 +15,7 @@ conda activate llm pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/ ``` -#### 1.2 Installation on Windows +### 1.2 Installation on Windows We suggest using conda to manage environment: ```bash conda create -n llm python=3.11 libuv @@ -25,7 +25,7 @@ conda activate llm pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/ ``` -### 2. Configures OneAPI environment variables for Linux +## 2. Configures OneAPI environment variables for Linux > [!NOTE] > Skip this step if you are running on Windows. @@ -36,9 +36,9 @@ This is a required step on Linux for APT or offline installed oneAPI. Skip this source /opt/intel/oneapi/setvars.sh ``` -### 3. Runtime Configurations +## 3. Runtime Configurations For optimal performance, it is recommended to set several environment variables. Please check out the suggestions based on your device. -#### 3.1 Configurations for Linux +### 3.1 Configurations for Linux
For Intel Arc™ A-Series Graphics and Intel Data Center GPU Flex Series @@ -75,7 +75,7 @@ export BIGDL_LLM_XMX_DISABLED=1
-#### 3.2 Configurations for Windows +### 3.2 Configurations for Windows
For Intel iGPU @@ -100,9 +100,9 @@ set SYCL_CACHE_PERSISTENT=1 > [!NOTE] > For the first time that each model runs on Intel iGPU/Intel Arc™ A300-Series or Pro A60, it may take several minutes to compile. -### 4. Running examples +## 4. Running examples -#### Example 1: Predict Tokens using `generate()` API +### Example 1: Predict Tokens using `generate()` API In the example [generate.py](./generate.py), we show a basic use case for a ChatGLM2 model to predict the next N tokens using `generate()` API, with IPEX-LLM INT4 optimizations on Intel GPUs. ``` @@ -141,7 +141,7 @@ Inference time: xxxx s 答： Artificial Intelligence (AI) refers to the ability of a computer or machine to perform tasks that typically require human-like intelligence, such as understanding language, recognizing patterns ``` -#### Example 2: Stream Chat using `stream_chat()` API +### Example 2: Stream Chat using `stream_chat()` API In the example [streamchat.py](./streamchat.py), we show a basic use case for a ChatGLM2 model to stream chat, with IPEX-LLM INT4 optimizations. **Stream Chat using `stream_chat()` API**: diff --git a/python/llm/example/GPU/HF-Transformers-AutoModels/Model/chatglm3/README.md b/python/llm/example/GPU/HF-Transformers-AutoModels/Model/chatglm3/README.md index f4179e24053..490da17f0d5 100644 --- a/python/llm/example/GPU/HF-Transformers-AutoModels/Model/chatglm3/README.md +++ b/python/llm/example/GPU/HF-Transformers-AutoModels/Model/chatglm3/README.md @@ -2,11 +2,11 @@ In this directory, you will find examples on how you could apply IPEX-LLM INT4 optimizations on ChatGLM3 models on [Intel GPUs](../../../README.md). For illustration purposes, we utilize the [THUDM/chatglm3-6b](https://huggingface.co/THUDM/chatglm3-6b) as a reference ChatGLM3 model. -### 0. Requirements +## 0. Requirements To run these examples with IPEX-LLM on Intel GPUs, we have some recommended requirements for your machine, please refer to [here](../../../README.md#requirements) for more information. -### 1. Install -#### 1.1 Installation on Linux +## 1. Install +### 1.1 Installation on Linux We suggest using conda to manage environment: ```bash conda create -n llm python=3.11 @@ -15,7 +15,7 @@ conda activate llm pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/ ``` -#### 1.2 Installation on Windows +### 1.2 Installation on Windows We suggest using conda to manage environment: ```bash conda create -n llm python=3.11 libuv @@ -25,7 +25,7 @@ conda activate llm pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/ ``` -### 2. Configures OneAPI environment variables for Linux +## 2. Configures OneAPI environment variables for Linux > [!NOTE] > Skip this step if you are running on Windows. @@ -36,9 +36,9 @@ This is a required step on Linux for APT or offline installed oneAPI. Skip this source /opt/intel/oneapi/setvars.sh ``` -### 3. Runtime Configurations +## 3. Runtime Configurations For optimal performance, it is recommended to set several environment variables. Please check out the suggestions based on your device. -#### 3.1 Configurations for Linux +### 3.1 Configurations for Linux
For Intel Arc™ A-Series Graphics and Intel Data Center GPU Flex Series @@ -75,7 +75,7 @@ export BIGDL_LLM_XMX_DISABLED=1
-#### 3.2 Configurations for Windows +### 3.2 Configurations for Windows
For Intel iGPU @@ -99,9 +99,9 @@ set SYCL_CACHE_PERSISTENT=1 > [!NOTE] > For the first time that each model runs on Intel iGPU/Intel Arc™ A300-Series or Pro A60, it may take several minutes to compile. -### 4. Running examples +## 4. Running examples -#### Example 1: Predict Tokens using `generate()` API +### Example 1: Predict Tokens using `generate()` API In the example [generate.py](./generate.py), we show a basic use case for a ChatGLM3 model to predict the next N tokens using `generate()` API, with IPEX-LLM INT4 optimizations on Intel GPUs. ``` @@ -141,7 +141,7 @@ What is AI? AI stands for Artificial Intelligence. It refers to the development of computer systems or machines that can perform tasks that would normally require human intelligence, such as recognizing patterns ``` -#### Example 2: Stream Chat using `stream_chat()` API +### Example 2: Stream Chat using `stream_chat()` API In the example [streamchat.py](./streamchat.py), we show a basic use case for a ChatGLM3 model to stream chat, with IPEX-LLM INT4 optimizations. **Stream Chat using `stream_chat()` API**: diff --git a/python/llm/example/GPU/HF-Transformers-AutoModels/Model/glm4/README.md b/python/llm/example/GPU/HF-Transformers-AutoModels/Model/glm4/README.md index 9ee248ca6c5..930e77af04b 100644 --- a/python/llm/example/GPU/HF-Transformers-AutoModels/Model/glm4/README.md +++ b/python/llm/example/GPU/HF-Transformers-AutoModels/Model/glm4/README.md @@ -1,11 +1,11 @@ # GLM-4 In this directory, you will find examples on how you could apply IPEX-LLM INT4 optimizations on GLM-4 models on [Intel GPUs](../../../README.md). For illustration purposes, we utilize the [THUDM/glm-4-9b-chat](https://huggingface.co/THUDM/glm-4-9b-chat) as a reference InternLM model. -### 0. Requirements +## 0. Requirements To run these examples with IPEX-LLM on Intel GPUs, we have some recommended requirements for your machine, please refer to [here](../../../README.md#requirements) for more information. -### 1. Install -#### 1.1 Installation on Linux +## 1. Install +### 1.1 Installation on Linux We suggest using conda to manage environment: ```bash conda create -n llm python=3.11 @@ -17,7 +17,7 @@ pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-exte pip install tiktoken ``` -#### 1.2 Installation on Windows +### 1.2 Installation on Windows We suggest using conda to manage environment: ```bash conda create -n llm python=3.11 libuv @@ -30,7 +30,7 @@ pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-exte pip install tiktoken ``` -### 2. Configures OneAPI environment variables for Linux +## 2. Configures OneAPI environment variables for Linux > [!NOTE] > Skip this step if you are running on Windows. @@ -41,9 +41,9 @@ This is a required step on Linux for APT or offline installed oneAPI. Skip this source /opt/intel/oneapi/setvars.sh ``` -### 3. Runtime Configurations +## 3. Runtime Configurations For optimal performance, it is recommended to set several environment variables. Please check out the suggestions based on your device. -#### 3.1 Configurations for Linux +### 3.1 Configurations for Linux
For Intel Arc™ A-Series Graphics and Intel Data Center GPU Flex Series @@ -80,7 +80,7 @@ export BIGDL_LLM_XMX_DISABLED=1
-#### 3.2 Configurations for Windows +### 3.2 Configurations for Windows
For Intel iGPU @@ -104,9 +104,9 @@ set SYCL_CACHE_PERSISTENT=1 > [!NOTE] > For the first time that each model runs on Intel iGPU/Intel Arc™ A300-Series or Pro A60, it may take several minutes to compile. -### 4. Running examples +## 4. Running examples -#### Example 1: Predict Tokens using `generate()` API +### Example 1: Predict Tokens using `generate()` API In the example [generate.py](./generate.py), we show a basic use case for a GLM-4 model to predict the next N tokens using `generate()` API, with IPEX-LLM INT4 optimizations on Intel GPUs. ``` @@ -147,7 +147,7 @@ What is AI? Artificial Intelligence (AI) refers to the simulation of human intelligence in machines that are programmed to think like humans and mimic their actions. The term "art ``` -#### Example 2: Stream Chat using `stream_chat()` API +### Example 2: Stream Chat using `stream_chat()` API In the example [streamchat.py](./streamchat.py), we show a basic use case for a GLM-4 model to stream chat, with IPEX-LLM INT4 optimizations. **Stream Chat using `stream_chat()` API**: diff --git a/python/llm/example/GPU/PyTorch-Models/Model/chatglm2/README.md b/python/llm/example/GPU/PyTorch-Models/Model/chatglm2/README.md index 43e6149df8a..070f754720e 100644 --- a/python/llm/example/GPU/PyTorch-Models/Model/chatglm2/README.md +++ b/python/llm/example/GPU/PyTorch-Models/Model/chatglm2/README.md @@ -4,8 +4,8 @@ In this directory, you will find examples on how you could use IPEX-LLM `optimiz ## Requirements To run these examples with IPEX-LLM on Intel GPUs, we have some recommended requirements for your machine, please refer to [here](../../../README.md#requirements) for more information. -### 1. Install -#### 1.1 Installation on Linux +## 1. Install +### 1.1 Installation on Linux We suggest using conda to manage environment: ```bash conda create -n llm python=3.11 @@ -14,7 +14,7 @@ conda activate llm pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/ ``` -#### 1.2 Installation on Windows +### 1.2 Installation on Windows We suggest using conda to manage environment: ```bash conda create -n llm python=3.11 libuv @@ -24,7 +24,7 @@ conda activate llm pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/ ``` -### 2. Configures OneAPI environment variables for Linux +## 2. Configures OneAPI environment variables for Linux > [!NOTE] > Skip this step if you are running on Windows. @@ -35,9 +35,9 @@ This is a required step on Linux for APT or offline installed oneAPI. Skip this source /opt/intel/oneapi/setvars.sh ``` -### 3. Runtime Configurations +## 3. Runtime Configurations For optimal performance, it is recommended to set several environment variables. Please check out the suggestions based on your device. -#### 3.1 Configurations for Linux +### 3.1 Configurations for Linux
For Intel Arc™ A-Series Graphics and Intel Data Center GPU Flex Series @@ -74,7 +74,7 @@ export BIGDL_LLM_XMX_DISABLED=1
-#### 3.2 Configurations for Windows +### 3.2 Configurations for Windows
For Intel iGPU @@ -99,9 +99,9 @@ set SYCL_CACHE_PERSISTENT=1 > [!NOTE] > For the first time that each model runs on Intel iGPU/Intel Arc™ A300-Series or Pro A60, it may take several minutes to compile. -### 4. Running examples +## 4. Running examples -#### Example 1: Predict Tokens using `generate()` API +### Example 1: Predict Tokens using `generate()` API In the example [generate.py](./generate.py), we show a basic use case for a ChatGLM2 model to predict the next N tokens using `generate()` API, with IPEX-LLM INT4 optimizations on Intel GPUs. ```bash @@ -136,7 +136,7 @@ Inference time: xxxx s 答： Artificial Intelligence (AI) refers to the ability of a computer or machine to perform tasks that typically require human-like intelligence, such as understanding language, recognizing patterns ``` -#### Example 2: Stream Chat using `stream_chat()` API +### Example 2: Stream Chat using `stream_chat()` API In the example [streamchat.py](./streamchat.py), we show a basic use case for a ChatGLM2 model to stream chat, with IPEX-LLM INT4 optimizations. **Stream Chat using `stream_chat()` API**: diff --git a/python/llm/example/GPU/PyTorch-Models/Model/chatglm3/README.md b/python/llm/example/GPU/PyTorch-Models/Model/chatglm3/README.md index fafe7aac795..335c8ea0762 100644 --- a/python/llm/example/GPU/PyTorch-Models/Model/chatglm3/README.md +++ b/python/llm/example/GPU/PyTorch-Models/Model/chatglm3/README.md @@ -4,8 +4,8 @@ In this directory, you will find examples on how you could use IPEX-LLM `optimiz ## Requirements To run these examples with IPEX-LLM on Intel GPUs, we have some recommended requirements for your machine, please refer to [here](../../../README.md#requirements) for more information. -### 1. Install -#### 1.1 Installation on Linux +## 1. Install +### 1.1 Installation on Linux We suggest using conda to manage environment: ```bash conda create -n llm python=3.11 @@ -14,7 +14,7 @@ conda activate llm pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/ ``` -#### 1.2 Installation on Windows +### 1.2 Installation on Windows We suggest using conda to manage environment: ```bash conda create -n llm python=3.11 libuv @@ -24,7 +24,7 @@ conda activate llm pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/ ``` -### 2. Configures OneAPI environment variables for Linux +## 2. Configures OneAPI environment variables for Linux > [!NOTE] > Skip this step if you are running on Windows. @@ -35,9 +35,9 @@ This is a required step on Linux for APT or offline installed oneAPI. Skip this source /opt/intel/oneapi/setvars.sh ``` -### 3. Runtime Configurations +## 3. Runtime Configurations For optimal performance, it is recommended to set several environment variables. Please check out the suggestions based on your device. -#### 3.1 Configurations for Linux +### 3.1 Configurations for Linux
For Intel Arc™ A-Series Graphics and Intel Data Center GPU Flex Series @@ -74,7 +74,7 @@ export BIGDL_LLM_XMX_DISABLED=1
-#### 3.2 Configurations for Windows +### 3.2 Configurations for Windows
For Intel iGPU @@ -98,9 +98,9 @@ set SYCL_CACHE_PERSISTENT=1 > [!NOTE] > For the first time that each model runs on Intel iGPU/Intel Arc™ A300-Series or Pro A60, it may take several minutes to compile. -### 4. Running examples +## 4. Running examples -#### Example 1: Predict Tokens using `generate()` API +### Example 1: Predict Tokens using `generate()` API In the example [generate.py](./generate.py), we show a basic use case for a ChatGLM3 model to predict the next N tokens using `generate()` API, with IPEX-LLM INT4 optimizations on Intel GPUs. ```bash @@ -133,7 +133,7 @@ What is AI? AI stands for Artificial Intelligence. It refers to the development of computer systems or machines that can perform tasks that would normally require human intelligence, such as recognizing patterns ``` -#### Example 2: Stream Chat using `stream_chat()` API +### Example 2: Stream Chat using `stream_chat()` API In the example [streamchat.py](./streamchat.py), we show a basic use case for a ChatGLM3 model to stream chat, with IPEX-LLM INT4 optimizations. **Stream Chat using `stream_chat()` API**: