From 1b70732042fefd5d0fb612c82edcba34e1acae74 Mon Sep 17 00:00:00 2001
From: jinbridge <2635480475@qq.com>
Date: Wed, 12 Jun 2024 16:12:49 +0800
Subject: [PATCH 1/4] merge install step in glm examples
---
.../Model/chatglm2/README.md | 44 ++-----
.../Model/chatglm3/README.md | 44 ++-----
.../Model/glm4/README.md | 49 ++------
.../Model/chatglm2/README.md | 110 ++---------------
.../Model/chatglm3/README.md | 109 ++---------------
.../Model/glm4/README.md | 115 ++----------------
.../PyTorch-Models/Model/chatglm2/README.md | 111 ++---------------
.../PyTorch-Models/Model/chatglm3/README.md | 106 ++--------------
8 files changed, 78 insertions(+), 610 deletions(-)
diff --git a/python/llm/example/CPU/HF-Transformers-AutoModels/Model/chatglm2/README.md b/python/llm/example/CPU/HF-Transformers-AutoModels/Model/chatglm2/README.md
index bcb96b93446..8efb927ba52 100644
--- a/python/llm/example/CPU/HF-Transformers-AutoModels/Model/chatglm2/README.md
+++ b/python/llm/example/CPU/HF-Transformers-AutoModels/Model/chatglm2/README.md
@@ -2,11 +2,9 @@
In this directory, you will find examples on how you could apply IPEX-LLM INT4 optimizations on ChatGLM2 models. For illustration purposes, we utilize the [THUDM/chatglm2-6b](https://huggingface.co/THUDM/chatglm2-6b) as a reference ChatGLM2 model.
-## 0. Requirements
+### 0. Requirements
To run these examples with IPEX-LLM, we have some recommended requirements for your machine, please refer to [here](../README.md#recommended-requirements) for more information.
-## Example 1: Predict Tokens using `generate()` API
-In the example [generate.py](./generate.py), we show a basic use case for a ChatGLM2 model to predict the next N tokens using `generate()` API, with IPEX-LLM INT4 optimizations.
### 1. Install
We suggest using conda to manage environment:
@@ -30,6 +28,10 @@ pip install --pre --upgrade ipex-llm[all]
```
### 2. Run
+
+#### Example 1: Predict Tokens using `generate()` API
+In the example [generate.py](./generate.py), we show a basic use case for a ChatGLM2 model to predict the next N tokens using `generate()` API, with IPEX-LLM INT4 optimizations.
+
```
python ./generate.py --repo-id-or-model-path REPO_ID_OR_MODEL_PATH --prompt PROMPT --n-predict N_PREDICT
```
@@ -43,13 +45,13 @@ Arguments info:
>
> Please select the appropriate size of the ChatGLM2 model based on the capabilities of your machine.
-#### 2.1 Client
+##### 2.1 Client
On client Windows machine, it is recommended to run directly with full utilization of all cores:
```cmd
python ./generate.py
```
-#### 2.2 Server
+##### 2.2 Server
For optimal performance on server, it is recommended to set several environment variables (refer to [here](../README.md#best-known-configuration-on-linux) for more information), and run the example with all the physical cores of a single socket.
E.g. on Linux,
@@ -62,8 +64,8 @@ export OMP_NUM_THREADS=48
numactl -C 0-47 -m 0 python ./generate.py
```
-#### 2.3 Sample Output
-#### [THUDM/chatglm2-6b](https://huggingface.co/THUDM/chatglm2-6b)
+##### 2.3 Sample Output
+##### [THUDM/chatglm2-6b](https://huggingface.co/THUDM/chatglm2-6b)
```log
Inference time: xxxx s
-------------------- Prompt --------------------
@@ -88,31 +90,9 @@ Inference time: xxxx s
答： Artificial Intelligence (AI) refers to the ability of a computer or machine to perform tasks that typically require human-like intelligence, such as understanding language, recognizing patterns
```
-## Example 2: Stream Chat using `stream_chat()` API
+#### Example 2: Stream Chat using `stream_chat()` API
In the example [streamchat.py](./streamchat.py), we show a basic use case for a ChatGLM2 model to stream chat, with IPEX-LLM INT4 optimizations.
-### 1. Install
-We suggest using conda to manage environment:
-
-On Linux:
-
-```bash
-conda create -n llm python=3.11 # recommend to use Python 3.11
-conda activate llm
-
-# install the latest ipex-llm nightly build with 'all' option
-pip install --pre --upgrade ipex-llm[all] --extra-index-url https://download.pytorch.org/whl/cpu
-```
-On Windows:
-
-```cmd
-conda create -n llm python=3.11
-conda activate llm
-
-pip install --pre --upgrade ipex-llm[all]
-```
-
-### 2. Run
**Stream Chat using `stream_chat()` API**:
```
python ./streamchat.py --repo-id-or-model-path REPO_ID_OR_MODEL_PATH --question QUESTION
@@ -132,14 +112,14 @@ Arguments info:
>
> Please select the appropriate size of the ChatGLM2 model based on the capabilities of your machine.
-#### 2.1 Client
+##### 2.1 Client
On client Windows machine, it is recommended to run directly with full utilization of all cores:
```cmd
$env:PYTHONUNBUFFERED=1 # ensure stdout and stderr streams are sent straight to terminal without being first buffered
python ./streamchat.py
```
-#### 2.2 Server
+##### 2.2 Server
For optimal performance on server, it is recommended to set several environment variables (refer to [here](../README.md#best-known-configuration-on-linux) for more information), and run the example with all the physical cores of a single socket.
E.g. on Linux,
diff --git a/python/llm/example/CPU/HF-Transformers-AutoModels/Model/chatglm3/README.md b/python/llm/example/CPU/HF-Transformers-AutoModels/Model/chatglm3/README.md
index 059fad28dd5..99dc26c0491 100644
--- a/python/llm/example/CPU/HF-Transformers-AutoModels/Model/chatglm3/README.md
+++ b/python/llm/example/CPU/HF-Transformers-AutoModels/Model/chatglm3/README.md
@@ -2,11 +2,9 @@
In this directory, you will find examples on how you could apply IPEX-LLM INT4 optimizations on ChatGLM3 models. For illustration purposes, we utilize the [THUDM/chatglm3-6b](https://huggingface.co/THUDM/chatglm3-6b) as a reference ChatGLM3 model.
-## 0. Requirements
+### 0. Requirements
To run these examples with IPEX-LLM, we have some recommended requirements for your machine, please refer to [here](../README.md#recommended-requirements) for more information.
-## Example 1: Predict Tokens using `generate()` API
-In the example [generate.py](./generate.py), we show a basic use case for a ChatGLM3 model to predict the next N tokens using `generate()` API, with IPEX-LLM INT4 optimizations.
### 1. Install
We suggest using conda to manage environment:
@@ -30,6 +28,10 @@ pip install --pre --upgrade ipex-llm[all]
```
### 2. Run
+
+#### Example 1: Predict Tokens using `generate()` API
+In the example [generate.py](./generate.py), we show a basic use case for a ChatGLM3 model to predict the next N tokens using `generate()` API, with IPEX-LLM INT4 optimizations.
+
```
python ./generate.py --repo-id-or-model-path REPO_ID_OR_MODEL_PATH --prompt PROMPT --n-predict N_PREDICT
```
@@ -43,13 +45,13 @@ Arguments info:
>
> Please select the appropriate size of the ChatGLM3 model based on the capabilities of your machine.
-#### 2.1 Client
+##### 2.1 Client
On client Windows machine, it is recommended to run directly with full utilization of all cores:
```cmd
python ./generate.py
```
-#### 2.2 Server
+##### 2.2 Server
For optimal performance on server, it is recommended to set several environment variables (refer to [here](../README.md#best-known-configuration-on-linux) for more information), and run the example with all the physical cores of a single socket.
E.g. on Linux,
@@ -62,8 +64,8 @@ export OMP_NUM_THREADS=48
numactl -C 0-47 -m 0 python ./generate.py
```
-#### 2.3 Sample Output
-#### [THUDM/chatglm3-6b](https://huggingface.co/THUDM/chatglm3-6b)
+##### 2.3 Sample Output
+##### [THUDM/chatglm3-6b](https://huggingface.co/THUDM/chatglm3-6b)
```log
Inference time: xxxx s
-------------------- Prompt --------------------
@@ -89,31 +91,9 @@ What is AI?
AI stands for Artificial Intelligence. It refers to the development of computer systems that can perform tasks that would normally require human intelligence, such as recognizing speech or making
```
-## Example 2: Stream Chat using `stream_chat()` API
+#### Example 2: Stream Chat using `stream_chat()` API
In the example [streamchat.py](./streamchat.py), we show a basic use case for a ChatGLM3 model to stream chat, with IPEX-LLM INT4 optimizations.
-### 1. Install
-We suggest using conda to manage environment:
-
-On Linux:
-
-```bash
-conda create -n llm python=3.11 # recommend to use Python 3.11
-conda activate llm
-
-# install the latest ipex-llm nightly build with 'all' option
-pip install --pre --upgrade ipex-llm[all] --extra-index-url https://download.pytorch.org/whl/cpu
-```
-On Windows:
-
-```cmd
-conda create -n llm python=3.11
-conda activate llm
-
-pip install --pre --upgrade ipex-llm[all]
-```
-
-### 2. Run
**Stream Chat using `stream_chat()` API**:
```
python ./streamchat.py --repo-id-or-model-path REPO_ID_OR_MODEL_PATH --question QUESTION
@@ -133,14 +113,14 @@ Arguments info:
>
> Please select the appropriate size of the ChatGLM3 model based on the capabilities of your machine.
-#### 2.1 Client
+##### 2.1 Client
On client Windows machine, it is recommended to run directly with full utilization of all cores:
```cmd
$env:PYTHONUNBUFFERED=1 # ensure stdout and stderr streams are sent straight to terminal without being first buffered
python ./streamchat.py
```
-#### 2.2 Server
+##### 2.2 Server
For optimal performance on server, it is recommended to set several environment variables (refer to [here](../README.md#best-known-configuration-on-linux) for more information), and run the example with all the physical cores of a single socket.
E.g. on Linux,
diff --git a/python/llm/example/CPU/HF-Transformers-AutoModels/Model/glm4/README.md b/python/llm/example/CPU/HF-Transformers-AutoModels/Model/glm4/README.md
index d3d2966159c..f62265f3d8a 100644
--- a/python/llm/example/CPU/HF-Transformers-AutoModels/Model/glm4/README.md
+++ b/python/llm/example/CPU/HF-Transformers-AutoModels/Model/glm4/README.md
@@ -2,11 +2,9 @@
In this directory, you will find examples on how you could apply IPEX-LLM INT4 optimizations on GLM-4 models. For illustration purposes, we utilize the [THUDM/glm-4-9b-chat](https://huggingface.co/THUDM/glm-4-9b-chat) as a reference GLM-4 model.
-## 0. Requirements
+### 0. Requirements
To run these examples with IPEX-LLM, we have some recommended requirements for your machine, please refer to [here](../README.md#recommended-requirements) for more information.
-## Example 1: Predict Tokens using `generate()` API
-In the example [generate.py](./generate.py), we show a basic use case for a GLM-4 model to predict the next N tokens using `generate()` API, with IPEX-LLM INT4 optimizations.
### 1. Install
We suggest using conda to manage environment:
@@ -35,6 +33,10 @@ pip install tiktoken
```
### 2. Run
+
+#### Example 1: Predict Tokens using `generate()` API
+In the example [generate.py](./generate.py), we show a basic use case for a GLM-4 model to predict the next N tokens using `generate()` API, with IPEX-LLM INT4 optimizations.
+
```
python ./generate.py --repo-id-or-model-path REPO_ID_OR_MODEL_PATH --prompt PROMPT --n-predict N_PREDICT
```
@@ -48,13 +50,13 @@ Arguments info:
>
> Please select the appropriate size of the GLM-4 model based on the capabilities of your machine.
-#### 2.1 Client
+##### 2.1 Client
On client Windows machine, it is recommended to run directly with full utilization of all cores:
```cmd
python ./generate.py
```
-#### 2.2 Server
+##### 2.2 Server
For optimal performance on server, it is recommended to set several environment variables (refer to [here](../README.md#best-known-configuration-on-linux) for more information), and run the example with all the physical cores of a single socket.
E.g. on Linux,
@@ -67,8 +69,8 @@ export OMP_NUM_THREADS=48
numactl -C 0-47 -m 0 python ./generate.py
```
-#### 2.3 Sample Output
-##### [THUDM/glm-4-9b-chat](https://huggingface.co/THUDM/glm-4-9b-chat)
+##### 2.3 Sample Output
+###### [THUDM/glm-4-9b-chat](https://huggingface.co/THUDM/glm-4-9b-chat)
```log
Inference time: xxxx s
-------------------- Prompt --------------------
@@ -95,36 +97,9 @@ What is AI?
Artificial Intelligence (AI) refers to the simulation of human intelligence in machines that are programmed to think like humans and mimic their actions. The term "art
```
-## Example 2: Stream Chat using `stream_chat()` API
+#### Example 2: Stream Chat using `stream_chat()` API
In the example [streamchat.py](./streamchat.py), we show a basic use case for a GLM-4 model to stream chat, with IPEX-LLM INT4 optimizations.
-### 1. Install
-We suggest using conda to manage environment:
-
-On Linux:
-```bash
-conda create -n llm python=3.11 # recommend to use Python 3.11
-conda activate llm
-
-# install the latest ipex-llm nightly build with 'all' option
-pip install --pre --upgrade ipex-llm[all] --extra-index-url https://download.pytorch.org/whl/cpu
-
-# install tiktoken required for GLM-4
-pip install tiktoken
-```
-
-On Windows:
-
-```cmd
-conda create -n llm python=3.11
-conda activate llm
-
-pip install --pre --upgrade ipex-llm[all]
-
-pip install tiktoken
-```
-
-### 2. Run
**Stream Chat using `stream_chat()` API**:
```
python ./streamchat.py --repo-id-or-model-path REPO_ID_OR_MODEL_PATH --question QUESTION
@@ -144,14 +119,14 @@ Arguments info:
>
> Please select the appropriate size of the GLM-4 model based on the capabilities of your machine.
-#### 2.1 Client
+##### 2.1 Client
On client Windows machine, it is recommended to run directly with full utilization of all cores:
```cmd
$env:PYTHONUNBUFFERED=1 # ensure stdout and stderr streams are sent straight to terminal without being first buffered
python ./streamchat.py
```
-#### 2.2 Server
+##### 2.2 Server
For optimal performance on server, it is recommended to set several environment variables (refer to [here](../README.md#best-known-configuration-on-linux) for more information), and run the example with all the physical cores of a single socket.
E.g. on Linux,
diff --git a/python/llm/example/GPU/HF-Transformers-AutoModels/Model/chatglm2/README.md b/python/llm/example/GPU/HF-Transformers-AutoModels/Model/chatglm2/README.md
index dab814eea0c..71bde904881 100644
--- a/python/llm/example/GPU/HF-Transformers-AutoModels/Model/chatglm2/README.md
+++ b/python/llm/example/GPU/HF-Transformers-AutoModels/Model/chatglm2/README.md
@@ -2,12 +2,9 @@
In this directory, you will find examples on how you could apply IPEX-LLM INT4 optimizations on ChatGLM2 models on [Intel GPUs](../../../README.md). For illustration purposes, we utilize the [THUDM/chatglm2-6b](https://huggingface.co/THUDM/chatglm2-6b) as a reference ChatGLM2 model.
-## 0. Requirements
+### 0. Requirements
To run these examples with IPEX-LLM on Intel GPUs, we have some recommended requirements for your machine, please refer to [here](../../../README.md#requirements) for more information.
-## Example 1: Predict Tokens using `generate()` API
-In the example [generate.py](./generate.py), we show a basic use case for a ChatGLM2 model to predict the next N tokens using `generate()` API, with IPEX-LLM INT4 optimizations on Intel GPUs.
-
### 1. Install
#### 1.1 Installation on Linux
We suggest using conda to manage environment:
@@ -104,6 +101,10 @@ set SYCL_CACHE_PERSISTENT=1
> For the first time that each model runs on Intel iGPU/Intel Arc™ A300-Series or Pro A60, it may take several minutes to compile.
### 4. Running examples
+
+#### Example 1: Predict Tokens using `generate()` API
+In the example [generate.py](./generate.py), we show a basic use case for a ChatGLM2 model to predict the next N tokens using `generate()` API, with IPEX-LLM INT4 optimizations on Intel GPUs.
+
```
python ./generate.py --repo-id-or-model-path REPO_ID_OR_MODEL_PATH --prompt PROMPT --n-predict N_PREDICT
```
@@ -113,8 +114,9 @@ Arguments info:
- `--prompt PROMPT`: argument defining the prompt to be infered (with integrated prompt format for chat). It is default to be `'AI是什么？'`.
- `--n-predict N_PREDICT`: argument defining the max number of tokens to predict. It is default to be `32`.
-#### Sample Output
-#### [THUDM/chatglm2-6b](https://huggingface.co/THUDM/chatglm2-6b)
+Sample Output:
+
+[THUDM/chatglm2-6b](https://huggingface.co/THUDM/chatglm2-6b)
```log
Inference time: xxxx s
-------------------- Prompt --------------------
@@ -139,103 +141,9 @@ Inference time: xxxx s
答： Artificial Intelligence (AI) refers to the ability of a computer or machine to perform tasks that typically require human-like intelligence, such as understanding language, recognizing patterns
```
-## Example 2: Stream Chat using `stream_chat()` API
+#### Example 2: Stream Chat using `stream_chat()` API
In the example [streamchat.py](./streamchat.py), we show a basic use case for a ChatGLM2 model to stream chat, with IPEX-LLM INT4 optimizations.
-### 1. Install
-#### 1.1 Installation on Linux
-We suggest using conda to manage environment:
-```bash
-conda create -n llm python=3.11
-conda activate llm
-# below command will install intel_extension_for_pytorch==2.1.10+xpu as default
-pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/
-```
-#### 1.2 Installation on Windows
-We suggest using conda to manage environment:
-```bash
-conda create -n llm python=3.11 libuv
-conda activate llm
-# below command will install intel_extension_for_pytorch==2.1.10+xpu as default
-pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/
-```
-
-
-### 2. Configures OneAPI environment variables for Linux
-
-> [!NOTE]
-> Skip this step if you are running on Windows.
-
-This is a required step on Linux for APT or offline installed oneAPI. Skip this step for PIP-installed oneAPI.
-
-```bash
-source /opt/intel/oneapi/setvars.sh
-```
-
-### 3. Runtime Configurations
-For optimal performance, it is recommended to set several environment variables. Please check out the suggestions based on your device.
-#### 3.1 Configurations for Linux
-
-
-For Intel Arc™ A-Series Graphics and Intel Data Center GPU Flex Series
-
-```bash
-export USE_XETLA=OFF
-export SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=1
-export SYCL_CACHE_PERSISTENT=1
-```
-
-
-
-
-For Intel Data Center GPU Max Series
-
-```bash
-export LD_PRELOAD=${LD_PRELOAD}:${CONDA_PREFIX}/lib/libtcmalloc.so
-export SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=1
-export SYCL_CACHE_PERSISTENT=1
-export ENABLE_SDP_FUSION=1
-```
-> Note: Please note that `libtcmalloc.so` can be installed by `conda install -c conda-forge -y gperftools=2.10`.
-
-
-
-
-For Intel iGPU
-
-```bash
-export SYCL_CACHE_PERSISTENT=1
-export BIGDL_LLM_XMX_DISABLED=1
-```
-
-
-
-#### 3.2 Configurations for Windows
-
-
-For Intel iGPU
-
-```cmd
-set SYCL_CACHE_PERSISTENT=1
-set BIGDL_LLM_XMX_DISABLED=1
-```
-
-
-
-
-
-For Intel Arc™ A-Series Graphics
-
-```cmd
-set SYCL_CACHE_PERSISTENT=1
-```
-
-
-
-> [!NOTE]
-> For the first time that each model runs on Intel iGPU/Intel Arc™ A300-Series or Pro A60, it may take several minutes to compile.
-
-### 4. Running examples
**Stream Chat using `stream_chat()` API**:
```
python ./streamchat.py --repo-id-or-model-path REPO_ID_OR_MODEL_PATH --question QUESTION
diff --git a/python/llm/example/GPU/HF-Transformers-AutoModels/Model/chatglm3/README.md b/python/llm/example/GPU/HF-Transformers-AutoModels/Model/chatglm3/README.md
index c4aca0e9977..f4179e24053 100644
--- a/python/llm/example/GPU/HF-Transformers-AutoModels/Model/chatglm3/README.md
+++ b/python/llm/example/GPU/HF-Transformers-AutoModels/Model/chatglm3/README.md
@@ -2,11 +2,9 @@
In this directory, you will find examples on how you could apply IPEX-LLM INT4 optimizations on ChatGLM3 models on [Intel GPUs](../../../README.md). For illustration purposes, we utilize the [THUDM/chatglm3-6b](https://huggingface.co/THUDM/chatglm3-6b) as a reference ChatGLM3 model.
-## 0. Requirements
+### 0. Requirements
To run these examples with IPEX-LLM on Intel GPUs, we have some recommended requirements for your machine, please refer to [here](../../../README.md#requirements) for more information.
-## Example 1: Predict Tokens using `generate()` API
-In the example [generate.py](./generate.py), we show a basic use case for a ChatGLM3 model to predict the next N tokens using `generate()` API, with IPEX-LLM INT4 optimizations on Intel GPUs.
### 1. Install
#### 1.1 Installation on Linux
We suggest using conda to manage environment:
@@ -103,6 +101,9 @@ set SYCL_CACHE_PERSISTENT=1
> For the first time that each model runs on Intel iGPU/Intel Arc™ A300-Series or Pro A60, it may take several minutes to compile.
### 4. Running examples
+#### Example 1: Predict Tokens using `generate()` API
+In the example [generate.py](./generate.py), we show a basic use case for a ChatGLM3 model to predict the next N tokens using `generate()` API, with IPEX-LLM INT4 optimizations on Intel GPUs.
+
```
python ./generate.py --repo-id-or-model-path REPO_ID_OR_MODEL_PATH --prompt PROMPT --n-predict N_PREDICT
```
@@ -112,8 +113,9 @@ Arguments info:
- `--prompt PROMPT`: argument defining the prompt to be infered (with integrated prompt format for chat). It is default to be `'AI是什么？'`.
- `--n-predict N_PREDICT`: argument defining the max number of tokens to predict. It is default to be `32`.
-#### Sample Output
-#### [THUDM/chatglm3-6b](https://huggingface.co/THUDM/chatglm3-6b)
+Sample Output:
+
+[THUDM/chatglm3-6b](https://huggingface.co/THUDM/chatglm3-6b)
```log
Inference time: xxxx s
-------------------- Prompt --------------------
@@ -139,103 +141,8 @@ What is AI?
AI stands for Artificial Intelligence. It refers to the development of computer systems or machines that can perform tasks that would normally require human intelligence, such as recognizing patterns
```
-## Example 2: Stream Chat using `stream_chat()` API
+#### Example 2: Stream Chat using `stream_chat()` API
In the example [streamchat.py](./streamchat.py), we show a basic use case for a ChatGLM3 model to stream chat, with IPEX-LLM INT4 optimizations.
-### 1. Install
-#### 1.1 Installation on Linux
-We suggest using conda to manage environment:
-```bash
-conda create -n llm python=3.11
-conda activate llm
-# below command will install intel_extension_for_pytorch==2.1.10+xpu as default
-pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/
-```
-
-#### 1.2 Installation on Windows
-We suggest using conda to manage environment:
-```bash
-conda create -n llm python=3.11 libuv
-conda activate llm
-
-# below command will install intel_extension_for_pytorch==2.1.10+xpu as default
-pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/
-```
-
-### 2. Configures OneAPI environment variables for Linux
-
-> [!NOTE]
-> Skip this step if you are running on Windows.
-
-This is a required step on Linux for APT or offline installed oneAPI. Skip this step for PIP-installed oneAPI.
-
-```bash
-source /opt/intel/oneapi/setvars.sh
-```
-
-### 3. Runtime Configurations
-For optimal performance, it is recommended to set several environment variables. Please check out the suggestions based on your device.
-#### 3.1 Configurations for Linux
-
-
-For Intel Arc™ A-Series Graphics and Intel Data Center GPU Flex Series
-
-```bash
-export USE_XETLA=OFF
-export SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=1
-export SYCL_CACHE_PERSISTENT=1
-```
-
-
-
-
-
-For Intel Data Center GPU Max Series
-
-```bash
-export LD_PRELOAD=${LD_PRELOAD}:${CONDA_PREFIX}/lib/libtcmalloc.so
-export SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=1
-export SYCL_CACHE_PERSISTENT=1
-export ENABLE_SDP_FUSION=1
-```
-> Note: Please note that `libtcmalloc.so` can be installed by `conda install -c conda-forge -y gperftools=2.10`.
-
-
-
-
-For Intel iGPU
-
-```bash
-export SYCL_CACHE_PERSISTENT=1
-export BIGDL_LLM_XMX_DISABLED=1
-```
-
-
-
-#### 3.2 Configurations for Windows
-
-
-For Intel iGPU
-
-```cmd
-set SYCL_CACHE_PERSISTENT=1
-set BIGDL_LLM_XMX_DISABLED=1
-```
-
-
-
-
-
-For Intel Arc™ A-Series Graphics
-
-```cmd
-set SYCL_CACHE_PERSISTENT=1
-```
-
-
-
-> [!NOTE]
-> For the first time that each model runs on Intel iGPU/Intel Arc™ A300-Series or Pro A60, it may take several minutes to compile.
-### 4. Running examples
**Stream Chat using `stream_chat()` API**:
```
diff --git a/python/llm/example/GPU/HF-Transformers-AutoModels/Model/glm4/README.md b/python/llm/example/GPU/HF-Transformers-AutoModels/Model/glm4/README.md
index 3d0c5d584a3..9ee248ca6c5 100644
--- a/python/llm/example/GPU/HF-Transformers-AutoModels/Model/glm4/README.md
+++ b/python/llm/example/GPU/HF-Transformers-AutoModels/Model/glm4/README.md
@@ -1,11 +1,9 @@
# GLM-4
In this directory, you will find examples on how you could apply IPEX-LLM INT4 optimizations on GLM-4 models on [Intel GPUs](../../../README.md). For illustration purposes, we utilize the [THUDM/glm-4-9b-chat](https://huggingface.co/THUDM/glm-4-9b-chat) as a reference InternLM model.
-## 0. Requirements
+### 0. Requirements
To run these examples with IPEX-LLM on Intel GPUs, we have some recommended requirements for your machine, please refer to [here](../../../README.md#requirements) for more information.
-## Example 1: Predict Tokens using `generate()` API
-In the example [generate.py](./generate.py), we show a basic use case for a GLM-4 model to predict the next N tokens using `generate()` API, with IPEX-LLM INT4 optimizations on Intel GPUs.
### 1. Install
#### 1.1 Installation on Linux
We suggest using conda to manage environment:
@@ -108,6 +106,9 @@ set SYCL_CACHE_PERSISTENT=1
> For the first time that each model runs on Intel iGPU/Intel Arc™ A300-Series or Pro A60, it may take several minutes to compile.
### 4. Running examples
+#### Example 1: Predict Tokens using `generate()` API
+In the example [generate.py](./generate.py), we show a basic use case for a GLM-4 model to predict the next N tokens using `generate()` API, with IPEX-LLM INT4 optimizations on Intel GPUs.
+
```
python ./generate.py --repo-id-or-model-path REPO_ID_OR_MODEL_PATH --prompt PROMPT --n-predict N_PREDICT
```
@@ -117,8 +118,9 @@ Arguments info:
- `--prompt PROMPT`: argument defining the prompt to be infered (with integrated prompt format for chat). It is default to be `'AI是什么？'`.
- `--n-predict N_PREDICT`: argument defining the max number of tokens to predict. It is default to be `32`.
-#### Sample Output
-##### [THUDM/glm-4-9b-chat](https://huggingface.co/THUDM/glm-4-9b-chat)
+Sample Output:
+
+[THUDM/glm-4-9b-chat](https://huggingface.co/THUDM/glm-4-9b-chat)
```log
Inference time: xxxx s
-------------------- Prompt --------------------
@@ -145,109 +147,8 @@ What is AI?
Artificial Intelligence (AI) refers to the simulation of human intelligence in machines that are programmed to think like humans and mimic their actions. The term "art
```
-## Example 2: Stream Chat using `stream_chat()` API
+#### Example 2: Stream Chat using `stream_chat()` API
In the example [streamchat.py](./streamchat.py), we show a basic use case for a GLM-4 model to stream chat, with IPEX-LLM INT4 optimizations.
-### 1. Install
-#### 1.1 Installation on Linux
-We suggest using conda to manage environment:
-```bash
-conda create -n llm python=3.11
-conda activate llm
-# below command will install intel_extension_for_pytorch==2.1.10+xpu as default
-pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/
-
-# install tiktoken required for GLM-4
-pip install tiktoken
-```
-
-#### 1.2 Installation on Windows
-We suggest using conda to manage environment:
-```bash
-conda create -n llm python=3.11 libuv
-conda activate llm
-
-# below command will install intel_extension_for_pytorch==2.1.10+xpu as default
-pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/
-
-# install tiktoken required for GLM-4
-pip install tiktoken
-```
-
-### 2. Configures OneAPI environment variables for Linux
-
-> [!NOTE]
-> Skip this step if you are running on Windows.
-
-This is a required step on Linux for APT or offline installed oneAPI. Skip this step for PIP-installed oneAPI.
-
-```bash
-source /opt/intel/oneapi/setvars.sh
-```
-
-### 3. Runtime Configurations
-For optimal performance, it is recommended to set several environment variables. Please check out the suggestions based on your device.
-#### 3.1 Configurations for Linux
-
-
-For Intel Arc™ A-Series Graphics and Intel Data Center GPU Flex Series
-
-```bash
-export USE_XETLA=OFF
-export SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=1
-export SYCL_CACHE_PERSISTENT=1
-```
-
-
-
-
-
-For Intel Data Center GPU Max Series
-
-```bash
-export LD_PRELOAD=${LD_PRELOAD}:${CONDA_PREFIX}/lib/libtcmalloc.so
-export SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=1
-export SYCL_CACHE_PERSISTENT=1
-export ENABLE_SDP_FUSION=1
-```
-> Note: Please note that `libtcmalloc.so` can be installed by `conda install -c conda-forge -y gperftools=2.10`.
-
-
-
-
-For Intel iGPU
-
-```bash
-export SYCL_CACHE_PERSISTENT=1
-export BIGDL_LLM_XMX_DISABLED=1
-```
-
-
-
-#### 3.2 Configurations for Windows
-
-
-For Intel iGPU
-
-```cmd
-set SYCL_CACHE_PERSISTENT=1
-set BIGDL_LLM_XMX_DISABLED=1
-```
-
-
-
-
-
-For Intel Arc™ A-Series Graphics
-
-```cmd
-set SYCL_CACHE_PERSISTENT=1
-```
-
-
-
-> [!NOTE]
-> For the first time that each model runs on Intel iGPU/Intel Arc™ A300-Series or Pro A60, it may take several minutes to compile.
-### 4. Running examples
**Stream Chat using `stream_chat()` API**:
```
diff --git a/python/llm/example/GPU/PyTorch-Models/Model/chatglm2/README.md b/python/llm/example/GPU/PyTorch-Models/Model/chatglm2/README.md
index ace1a38b167..43e6149df8a 100644
--- a/python/llm/example/GPU/PyTorch-Models/Model/chatglm2/README.md
+++ b/python/llm/example/GPU/PyTorch-Models/Model/chatglm2/README.md
@@ -4,8 +4,6 @@ In this directory, you will find examples on how you could use IPEX-LLM `optimiz
## Requirements
To run these examples with IPEX-LLM on Intel GPUs, we have some recommended requirements for your machine, please refer to [here](../../../README.md#requirements) for more information.
-## Example 1: Predict Tokens using `generate()` API
-In the example [generate.py](./generate.py), we show a basic use case for a ChatGLM2 model to predict the next N tokens using `generate()` API, with IPEX-LLM INT4 optimizations on Intel GPUs.
### 1. Install
#### 1.1 Installation on Linux
We suggest using conda to manage environment:
@@ -100,8 +98,12 @@ set SYCL_CACHE_PERSISTENT=1
> [!NOTE]
> For the first time that each model runs on Intel iGPU/Intel Arc™ A300-Series or Pro A60, it may take several minutes to compile.
+
### 4. Running examples
+#### Example 1: Predict Tokens using `generate()` API
+In the example [generate.py](./generate.py), we show a basic use case for a ChatGLM2 model to predict the next N tokens using `generate()` API, with IPEX-LLM INT4 optimizations on Intel GPUs.
+
```bash
python ./generate.py --prompt 'AI是什么？'
```
@@ -112,8 +114,10 @@ In the example, several arguments can be passed to satisfy your requirements:
- `--prompt PROMPT`: argument defining the prompt to be infered (with integrated prompt format for chat). It is default to be `'AI是什么？'`.
- `--n-predict N_PREDICT`: argument defining the max number of tokens to predict. It is default to be `32`.
-#### 4.1 Sample Output
-#### [THUDM/chatglm2-6b](https://huggingface.co/THUDM/chatglm2-6b)
+Sample Output:
+
+[THUDM/chatglm2-6b](https://huggingface.co/THUDM/chatglm2-6b)
+
```log
Inference time: xxxx s
-------------------- Output --------------------
@@ -132,103 +136,8 @@ Inference time: xxxx s
答： Artificial Intelligence (AI) refers to the ability of a computer or machine to perform tasks that typically require human-like intelligence, such as understanding language, recognizing patterns
```
-## Example 2: Stream Chat using `stream_chat()` API
+#### Example 2: Stream Chat using `stream_chat()` API
In the example [streamchat.py](./streamchat.py), we show a basic use case for a ChatGLM2 model to stream chat, with IPEX-LLM INT4 optimizations.
-### 1. Install
-#### 1.1 Installation on Linux
-We suggest using conda to manage environment:
-```bash
-conda create -n llm python=3.11
-conda activate llm
-# below command will install intel_extension_for_pytorch==2.1.10+xpu as default
-pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/
-```
-
-#### 1.2 Installation on Windows
-We suggest using conda to manage environment:
-```bash
-conda create -n llm python=3.11 libuv
-conda activate llm
-
-# below command will install intel_extension_for_pytorch==2.1.10+xpu as default
-pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/
-```
-
-### 2. Configures OneAPI environment variables for Linux
-
-> [!NOTE]
-> Skip this step if you are running on Windows.
-
-This is a required step on Linux for APT or offline installed oneAPI. Skip this step for PIP-installed oneAPI.
-
-```bash
-source /opt/intel/oneapi/setvars.sh
-```
-
-### 3. Runtime Configurations
-For optimal performance, it is recommended to set several environment variables. Please check out the suggestions based on your device.
-#### 3.1 Configurations for Linux
-
-
-For Intel Arc™ A-Series Graphics and Intel Data Center GPU Flex Series
-
-```bash
-export USE_XETLA=OFF
-export SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=1
-export SYCL_CACHE_PERSISTENT=1
-```
-
-
-
-
-
-For Intel Data Center GPU Max Series
-
-```bash
-export LD_PRELOAD=${LD_PRELOAD}:${CONDA_PREFIX}/lib/libtcmalloc.so
-export SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=1
-export SYCL_CACHE_PERSISTENT=1
-export ENABLE_SDP_FUSION=1
-```
-> Note: Please note that `libtcmalloc.so` can be installed by `conda install -c conda-forge -y gperftools=2.10`.
-
-
-
-
-For Intel iGPU
-
-```bash
-export SYCL_CACHE_PERSISTENT=1
-export BIGDL_LLM_XMX_DISABLED=1
-```
-
-
-
-#### 3.2 Configurations for Windows
-
-
-For Intel iGPU
-
-```cmd
-set SYCL_CACHE_PERSISTENT=1
-set BIGDL_LLM_XMX_DISABLED=1
-```
-
-
-
-
-
-For Intel Arc™ A-Series Graphics
-
-```cmd
-set SYCL_CACHE_PERSISTENT=1
-```
-
-
-
-> [!NOTE]
-> For the first time that each model runs on Intel iGPU/Intel Arc™ A300-Series or Pro A60, it may take several minutes to compile.
-### 4. Running examples
**Stream Chat using `stream_chat()` API**:
```
@@ -244,4 +153,4 @@ In the example, several arguments can be passed to satisfy your requirements:
- `--repo-id-or-model-path REPO_ID_OR_MODEL_PATH`: argument defining the huggingface repo id for the ChatGLM2 model to be downloaded, or the path to the huggingface checkpoint folder. It is default to be `'THUDM/chatglm2-6b'`.
- `--question QUESTION`: argument defining the question to ask. It is default to be `"晚上睡不着应该怎么办"`.
-- `--disable-stream`: argument defining whether to stream chat. If include `--disable-stream` when running the script, the stream chat is disabled and `chat()` API is used.
\ No newline at end of file
+- `--disable-stream`: argument defining whether to stream chat. If include `--disable-stream` when running the script, the stream chat is disabled and `chat()` API is used.
diff --git a/python/llm/example/GPU/PyTorch-Models/Model/chatglm3/README.md b/python/llm/example/GPU/PyTorch-Models/Model/chatglm3/README.md
index e85d070a8e2..fafe7aac795 100644
--- a/python/llm/example/GPU/PyTorch-Models/Model/chatglm3/README.md
+++ b/python/llm/example/GPU/PyTorch-Models/Model/chatglm3/README.md
@@ -4,8 +4,6 @@ In this directory, you will find examples on how you could use IPEX-LLM `optimiz
## Requirements
To run these examples with IPEX-LLM on Intel GPUs, we have some recommended requirements for your machine, please refer to [here](../../../README.md#requirements) for more information.
-## Example 1: Predict Tokens using `generate()` API
-In the example [generate.py](./generate.py), we show a basic use case for a ChatGLM3 model to predict the next N tokens using `generate()` API, with IPEX-LLM INT4 optimizations on Intel GPUs.
### 1. Install
#### 1.1 Installation on Linux
We suggest using conda to manage environment:
@@ -102,6 +100,9 @@ set SYCL_CACHE_PERSISTENT=1
> For the first time that each model runs on Intel iGPU/Intel Arc™ A300-Series or Pro A60, it may take several minutes to compile.
### 4. Running examples
+#### Example 1: Predict Tokens using `generate()` API
+In the example [generate.py](./generate.py), we show a basic use case for a ChatGLM3 model to predict the next N tokens using `generate()` API, with IPEX-LLM INT4 optimizations on Intel GPUs.
+
```bash
python ./generate.py --prompt 'AI是什么？'
```
@@ -112,8 +113,9 @@ In the example, several arguments can be passed to satisfy your requirements:
- `--prompt PROMPT`: argument defining the prompt to be infered (with integrated prompt format for chat). It is default to be `'AI是什么？'`.
- `--n-predict N_PREDICT`: argument defining the max number of tokens to predict. It is default to be `32`.
-#### 4.1 Sample Output
-#### [THUDM/chatglm3-6b](https://huggingface.co/THUDM/chatglm3-6b)
+Sample Output:
+
+[THUDM/chatglm3-6b](https://huggingface.co/THUDM/chatglm3-6b)
```log
Inference time: xxxx s
-------------------- Output --------------------
@@ -131,103 +133,9 @@ What is AI?
AI stands for Artificial Intelligence. It refers to the development of computer systems or machines that can perform tasks that would normally require human intelligence, such as recognizing patterns
```
-## Example 2: Stream Chat using `stream_chat()` API
+#### Example 2: Stream Chat using `stream_chat()` API
In the example [streamchat.py](./streamchat.py), we show a basic use case for a ChatGLM3 model to stream chat, with IPEX-LLM INT4 optimizations.
-### 1. Install
-#### 1.1 Installation on Linux
-We suggest using conda to manage environment:
-```bash
-conda create -n llm python=3.11
-conda activate llm
-# below command will install intel_extension_for_pytorch==2.1.10+xpu as default
-pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/
-```
-
-#### 1.2 Installation on Windows
-We suggest using conda to manage environment:
-```bash
-conda create -n llm python=3.11 libuv
-conda activate llm
-
-# below command will install intel_extension_for_pytorch==2.1.10+xpu as default
-pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/
-```
-
-### 2. Configures OneAPI environment variables for Linux
-
-> [!NOTE]
-> Skip this step if you are running on Windows.
-
-This is a required step on Linux for APT or offline installed oneAPI. Skip this step for PIP-installed oneAPI.
-
-```bash
-source /opt/intel/oneapi/setvars.sh
-```
-
-### 3. Runtime Configurations
-For optimal performance, it is recommended to set several environment variables. Please check out the suggestions based on your device.
-#### 3.1 Configurations for Linux
-
-
-For Intel Arc™ A-Series Graphics and Intel Data Center GPU Flex Series
-
-```bash
-export USE_XETLA=OFF
-export SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=1
-export SYCL_CACHE_PERSISTENT=1
-```
-
-
-
-
-For Intel Data Center GPU Max Series
-
-```bash
-export LD_PRELOAD=${LD_PRELOAD}:${CONDA_PREFIX}/lib/libtcmalloc.so
-export SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=1
-export SYCL_CACHE_PERSISTENT=1
-export ENABLE_SDP_FUSION=1
-```
-> Note: Please note that `libtcmalloc.so` can be installed by `conda install -c conda-forge -y gperftools=2.10`.
-
-
-
-
-For Intel iGPU
-
-```bash
-export SYCL_CACHE_PERSISTENT=1
-export BIGDL_LLM_XMX_DISABLED=1
-```
-
-
-
-#### 3.2 Configurations for Windows
-
-
-For Intel iGPU
-
-```cmd
-set SYCL_CACHE_PERSISTENT=1
-set BIGDL_LLM_XMX_DISABLED=1
-```
-
-
-
-
-
-For Intel Arc™ A-Series Graphics
-
-```cmd
-set SYCL_CACHE_PERSISTENT=1
-```
-
-
-
-> [!NOTE]
-> For the first time that each model runs on Intel iGPU/Intel Arc™ A300-Series or Pro A60, it may take several minutes to compile.
-### 4. Running examples
**Stream Chat using `stream_chat()` API**:
```
python ./streamchat.py
From 8d99c98cff6cad6060b665cf89e374b1ba267e8b Mon Sep 17 00:00:00 2001
From: jinbridge <2635480475@qq.com>
Date: Thu, 13 Jun 2024 15:20:52 +0800
Subject: [PATCH 2/4] fix section
---
.../Model/chatglm2/README.md | 20 ++++++++---------
.../Model/chatglm3/README.md | 20 ++++++++---------
.../Model/glm4/README.md | 22 +++++++++----------
.../Model/chatglm2/README.md | 22 +++++++++----------
.../Model/chatglm3/README.md | 22 +++++++++----------
.../Model/glm4/README.md | 22 +++++++++----------
.../PyTorch-Models/Model/chatglm2/README.md | 20 ++++++++---------
.../PyTorch-Models/Model/chatglm3/README.md | 20 ++++++++---------
8 files changed, 84 insertions(+), 84 deletions(-)
diff --git a/python/llm/example/CPU/HF-Transformers-AutoModels/Model/chatglm2/README.md b/python/llm/example/CPU/HF-Transformers-AutoModels/Model/chatglm2/README.md
index 8efb927ba52..bc9a5bf6d9d 100644
--- a/python/llm/example/CPU/HF-Transformers-AutoModels/Model/chatglm2/README.md
+++ b/python/llm/example/CPU/HF-Transformers-AutoModels/Model/chatglm2/README.md
@@ -2,10 +2,10 @@
In this directory, you will find examples on how you could apply IPEX-LLM INT4 optimizations on ChatGLM2 models. For illustration purposes, we utilize the [THUDM/chatglm2-6b](https://huggingface.co/THUDM/chatglm2-6b) as a reference ChatGLM2 model.
-### 0. Requirements
+## 0. Requirements
To run these examples with IPEX-LLM, we have some recommended requirements for your machine, please refer to [here](../README.md#recommended-requirements) for more information.
-### 1. Install
+## 1. Install
We suggest using conda to manage environment:
On Linux:
@@ -27,9 +27,9 @@ conda activate llm
pip install --pre --upgrade ipex-llm[all]
```
-### 2. Run
+## 2. Run
-#### Example 1: Predict Tokens using `generate()` API
+### Example 1: Predict Tokens using `generate()` API
In the example [generate.py](./generate.py), we show a basic use case for a ChatGLM2 model to predict the next N tokens using `generate()` API, with IPEX-LLM INT4 optimizations.
```
@@ -45,13 +45,13 @@ Arguments info:
>
> Please select the appropriate size of the ChatGLM2 model based on the capabilities of your machine.
-##### 2.1 Client
+#### 2.1 Client
On client Windows machine, it is recommended to run directly with full utilization of all cores:
```cmd
python ./generate.py
```
-##### 2.2 Server
+#### 2.2 Server
For optimal performance on server, it is recommended to set several environment variables (refer to [here](../README.md#best-known-configuration-on-linux) for more information), and run the example with all the physical cores of a single socket.
E.g. on Linux,
@@ -64,7 +64,7 @@ export OMP_NUM_THREADS=48
numactl -C 0-47 -m 0 python ./generate.py
```
-##### 2.3 Sample Output
+#### 2.3 Sample Output
##### [THUDM/chatglm2-6b](https://huggingface.co/THUDM/chatglm2-6b)
```log
Inference time: xxxx s
@@ -90,7 +90,7 @@ Inference time: xxxx s
答： Artificial Intelligence (AI) refers to the ability of a computer or machine to perform tasks that typically require human-like intelligence, such as understanding language, recognizing patterns
```
-#### Example 2: Stream Chat using `stream_chat()` API
+### Example 2: Stream Chat using `stream_chat()` API
In the example [streamchat.py](./streamchat.py), we show a basic use case for a ChatGLM2 model to stream chat, with IPEX-LLM INT4 optimizations.
**Stream Chat using `stream_chat()` API**:
@@ -112,14 +112,14 @@ Arguments info:
>
> Please select the appropriate size of the ChatGLM2 model based on the capabilities of your machine.
-##### 2.1 Client
+#### 2.1 Client
On client Windows machine, it is recommended to run directly with full utilization of all cores:
```cmd
$env:PYTHONUNBUFFERED=1 # ensure stdout and stderr streams are sent straight to terminal without being first buffered
python ./streamchat.py
```
-##### 2.2 Server
+#### 2.2 Server
For optimal performance on server, it is recommended to set several environment variables (refer to [here](../README.md#best-known-configuration-on-linux) for more information), and run the example with all the physical cores of a single socket.
E.g. on Linux,
diff --git a/python/llm/example/CPU/HF-Transformers-AutoModels/Model/chatglm3/README.md b/python/llm/example/CPU/HF-Transformers-AutoModels/Model/chatglm3/README.md
index 99dc26c0491..5a14aec47d7 100644
--- a/python/llm/example/CPU/HF-Transformers-AutoModels/Model/chatglm3/README.md
+++ b/python/llm/example/CPU/HF-Transformers-AutoModels/Model/chatglm3/README.md
@@ -2,10 +2,10 @@
In this directory, you will find examples on how you could apply IPEX-LLM INT4 optimizations on ChatGLM3 models. For illustration purposes, we utilize the [THUDM/chatglm3-6b](https://huggingface.co/THUDM/chatglm3-6b) as a reference ChatGLM3 model.
-### 0. Requirements
+## 0. Requirements
To run these examples with IPEX-LLM, we have some recommended requirements for your machine, please refer to [here](../README.md#recommended-requirements) for more information.
-### 1. Install
+## 1. Install
We suggest using conda to manage environment:
On Linux:
@@ -27,9 +27,9 @@ conda activate llm
pip install --pre --upgrade ipex-llm[all]
```
-### 2. Run
+## 2. Run
-#### Example 1: Predict Tokens using `generate()` API
+### Example 1: Predict Tokens using `generate()` API
In the example [generate.py](./generate.py), we show a basic use case for a ChatGLM3 model to predict the next N tokens using `generate()` API, with IPEX-LLM INT4 optimizations.
```
@@ -45,13 +45,13 @@ Arguments info:
>
> Please select the appropriate size of the ChatGLM3 model based on the capabilities of your machine.
-##### 2.1 Client
+#### 2.1 Client
On client Windows machine, it is recommended to run directly with full utilization of all cores:
```cmd
python ./generate.py
```
-##### 2.2 Server
+#### 2.2 Server
For optimal performance on server, it is recommended to set several environment variables (refer to [here](../README.md#best-known-configuration-on-linux) for more information), and run the example with all the physical cores of a single socket.
E.g. on Linux,
@@ -64,7 +64,7 @@ export OMP_NUM_THREADS=48
numactl -C 0-47 -m 0 python ./generate.py
```
-##### 2.3 Sample Output
+#### 2.3 Sample Output
##### [THUDM/chatglm3-6b](https://huggingface.co/THUDM/chatglm3-6b)
```log
Inference time: xxxx s
@@ -91,7 +91,7 @@ What is AI?
AI stands for Artificial Intelligence. It refers to the development of computer systems that can perform tasks that would normally require human intelligence, such as recognizing speech or making
```
-#### Example 2: Stream Chat using `stream_chat()` API
+### Example 2: Stream Chat using `stream_chat()` API
In the example [streamchat.py](./streamchat.py), we show a basic use case for a ChatGLM3 model to stream chat, with IPEX-LLM INT4 optimizations.
**Stream Chat using `stream_chat()` API**:
@@ -113,14 +113,14 @@ Arguments info:
>
> Please select the appropriate size of the ChatGLM3 model based on the capabilities of your machine.
-##### 2.1 Client
+#### 2.1 Client
On client Windows machine, it is recommended to run directly with full utilization of all cores:
```cmd
$env:PYTHONUNBUFFERED=1 # ensure stdout and stderr streams are sent straight to terminal without being first buffered
python ./streamchat.py
```
-##### 2.2 Server
+#### 2.2 Server
For optimal performance on server, it is recommended to set several environment variables (refer to [here](../README.md#best-known-configuration-on-linux) for more information), and run the example with all the physical cores of a single socket.
E.g. on Linux,
diff --git a/python/llm/example/CPU/HF-Transformers-AutoModels/Model/glm4/README.md b/python/llm/example/CPU/HF-Transformers-AutoModels/Model/glm4/README.md
index f62265f3d8a..db2a0c524bb 100644
--- a/python/llm/example/CPU/HF-Transformers-AutoModels/Model/glm4/README.md
+++ b/python/llm/example/CPU/HF-Transformers-AutoModels/Model/glm4/README.md
@@ -2,10 +2,10 @@
In this directory, you will find examples on how you could apply IPEX-LLM INT4 optimizations on GLM-4 models. For illustration purposes, we utilize the [THUDM/glm-4-9b-chat](https://huggingface.co/THUDM/glm-4-9b-chat) as a reference GLM-4 model.
-### 0. Requirements
+## 0. Requirements
To run these examples with IPEX-LLM, we have some recommended requirements for your machine, please refer to [here](../README.md#recommended-requirements) for more information.
-### 1. Install
+## 1. Install
We suggest using conda to manage environment:
On Linux:
@@ -32,9 +32,9 @@ pip install --pre --upgrade ipex-llm[all]
pip install tiktoken
```
-### 2. Run
+## 2. Run
-#### Example 1: Predict Tokens using `generate()` API
+### Example 1: Predict Tokens using `generate()` API
In the example [generate.py](./generate.py), we show a basic use case for a GLM-4 model to predict the next N tokens using `generate()` API, with IPEX-LLM INT4 optimizations.
```
@@ -50,13 +50,13 @@ Arguments info:
>
> Please select the appropriate size of the GLM-4 model based on the capabilities of your machine.
-##### 2.1 Client
+#### 2.1 Client
On client Windows machine, it is recommended to run directly with full utilization of all cores:
```cmd
python ./generate.py
```
-##### 2.2 Server
+#### 2.2 Server
For optimal performance on server, it is recommended to set several environment variables (refer to [here](../README.md#best-known-configuration-on-linux) for more information), and run the example with all the physical cores of a single socket.
E.g. on Linux,
@@ -69,8 +69,8 @@ export OMP_NUM_THREADS=48
numactl -C 0-47 -m 0 python ./generate.py
```
-##### 2.3 Sample Output
-###### [THUDM/glm-4-9b-chat](https://huggingface.co/THUDM/glm-4-9b-chat)
+#### 2.3 Sample Output
+##### [THUDM/glm-4-9b-chat](https://huggingface.co/THUDM/glm-4-9b-chat)
```log
Inference time: xxxx s
-------------------- Prompt --------------------
@@ -97,7 +97,7 @@ What is AI?
Artificial Intelligence (AI) refers to the simulation of human intelligence in machines that are programmed to think like humans and mimic their actions. The term "art
```
-#### Example 2: Stream Chat using `stream_chat()` API
+### Example 2: Stream Chat using `stream_chat()` API
In the example [streamchat.py](./streamchat.py), we show a basic use case for a GLM-4 model to stream chat, with IPEX-LLM INT4 optimizations.
**Stream Chat using `stream_chat()` API**:
@@ -119,14 +119,14 @@ Arguments info:
>
> Please select the appropriate size of the GLM-4 model based on the capabilities of your machine.
-##### 2.1 Client
+#### 2.1 Client
On client Windows machine, it is recommended to run directly with full utilization of all cores:
```cmd
$env:PYTHONUNBUFFERED=1 # ensure stdout and stderr streams are sent straight to terminal without being first buffered
python ./streamchat.py
```
-##### 2.2 Server
+#### 2.2 Server
For optimal performance on server, it is recommended to set several environment variables (refer to [here](../README.md#best-known-configuration-on-linux) for more information), and run the example with all the physical cores of a single socket.
E.g. on Linux,
diff --git a/python/llm/example/GPU/HF-Transformers-AutoModels/Model/chatglm2/README.md b/python/llm/example/GPU/HF-Transformers-AutoModels/Model/chatglm2/README.md
index 71bde904881..9f6a268a30e 100644
--- a/python/llm/example/GPU/HF-Transformers-AutoModels/Model/chatglm2/README.md
+++ b/python/llm/example/GPU/HF-Transformers-AutoModels/Model/chatglm2/README.md
@@ -2,11 +2,11 @@
In this directory, you will find examples on how you could apply IPEX-LLM INT4 optimizations on ChatGLM2 models on [Intel GPUs](../../../README.md). For illustration purposes, we utilize the [THUDM/chatglm2-6b](https://huggingface.co/THUDM/chatglm2-6b) as a reference ChatGLM2 model.
-### 0. Requirements
+## 0. Requirements
To run these examples with IPEX-LLM on Intel GPUs, we have some recommended requirements for your machine, please refer to [here](../../../README.md#requirements) for more information.
-### 1. Install
-#### 1.1 Installation on Linux
+## 1. Install
+### 1.1 Installation on Linux
We suggest using conda to manage environment:
```bash
conda create -n llm python=3.11
@@ -15,7 +15,7 @@ conda activate llm
pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/
```
-#### 1.2 Installation on Windows
+### 1.2 Installation on Windows
We suggest using conda to manage environment:
```bash
conda create -n llm python=3.11 libuv
@@ -25,7 +25,7 @@ conda activate llm
pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/
```
-### 2. Configures OneAPI environment variables for Linux
+## 2. Configures OneAPI environment variables for Linux
> [!NOTE]
> Skip this step if you are running on Windows.
@@ -36,9 +36,9 @@ This is a required step on Linux for APT or offline installed oneAPI. Skip this
source /opt/intel/oneapi/setvars.sh
```
-### 3. Runtime Configurations
+## 3. Runtime Configurations
For optimal performance, it is recommended to set several environment variables. Please check out the suggestions based on your device.
-#### 3.1 Configurations for Linux
+### 3.1 Configurations for Linux
For Intel Arc™ A-Series Graphics and Intel Data Center GPU Flex Series
@@ -75,7 +75,7 @@ export BIGDL_LLM_XMX_DISABLED=1
-#### 3.2 Configurations for Windows
+### 3.2 Configurations for Windows
For Intel iGPU
@@ -100,9 +100,9 @@ set SYCL_CACHE_PERSISTENT=1
> [!NOTE]
> For the first time that each model runs on Intel iGPU/Intel Arc™ A300-Series or Pro A60, it may take several minutes to compile.
-### 4. Running examples
+## 4. Running examples
-#### Example 1: Predict Tokens using `generate()` API
+### Example 1: Predict Tokens using `generate()` API
In the example [generate.py](./generate.py), we show a basic use case for a ChatGLM2 model to predict the next N tokens using `generate()` API, with IPEX-LLM INT4 optimizations on Intel GPUs.
```
@@ -141,7 +141,7 @@ Inference time: xxxx s
答： Artificial Intelligence (AI) refers to the ability of a computer or machine to perform tasks that typically require human-like intelligence, such as understanding language, recognizing patterns
```
-#### Example 2: Stream Chat using `stream_chat()` API
+### Example 2: Stream Chat using `stream_chat()` API
In the example [streamchat.py](./streamchat.py), we show a basic use case for a ChatGLM2 model to stream chat, with IPEX-LLM INT4 optimizations.
**Stream Chat using `stream_chat()` API**:
diff --git a/python/llm/example/GPU/HF-Transformers-AutoModels/Model/chatglm3/README.md b/python/llm/example/GPU/HF-Transformers-AutoModels/Model/chatglm3/README.md
index f4179e24053..490da17f0d5 100644
--- a/python/llm/example/GPU/HF-Transformers-AutoModels/Model/chatglm3/README.md
+++ b/python/llm/example/GPU/HF-Transformers-AutoModels/Model/chatglm3/README.md
@@ -2,11 +2,11 @@
In this directory, you will find examples on how you could apply IPEX-LLM INT4 optimizations on ChatGLM3 models on [Intel GPUs](../../../README.md). For illustration purposes, we utilize the [THUDM/chatglm3-6b](https://huggingface.co/THUDM/chatglm3-6b) as a reference ChatGLM3 model.
-### 0. Requirements
+## 0. Requirements
To run these examples with IPEX-LLM on Intel GPUs, we have some recommended requirements for your machine, please refer to [here](../../../README.md#requirements) for more information.
-### 1. Install
-#### 1.1 Installation on Linux
+## 1. Install
+### 1.1 Installation on Linux
We suggest using conda to manage environment:
```bash
conda create -n llm python=3.11
@@ -15,7 +15,7 @@ conda activate llm
pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/
```
-#### 1.2 Installation on Windows
+### 1.2 Installation on Windows
We suggest using conda to manage environment:
```bash
conda create -n llm python=3.11 libuv
@@ -25,7 +25,7 @@ conda activate llm
pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/
```
-### 2. Configures OneAPI environment variables for Linux
+## 2. Configures OneAPI environment variables for Linux
> [!NOTE]
> Skip this step if you are running on Windows.
@@ -36,9 +36,9 @@ This is a required step on Linux for APT or offline installed oneAPI. Skip this
source /opt/intel/oneapi/setvars.sh
```
-### 3. Runtime Configurations
+## 3. Runtime Configurations
For optimal performance, it is recommended to set several environment variables. Please check out the suggestions based on your device.
-#### 3.1 Configurations for Linux
+### 3.1 Configurations for Linux
For Intel Arc™ A-Series Graphics and Intel Data Center GPU Flex Series
@@ -75,7 +75,7 @@ export BIGDL_LLM_XMX_DISABLED=1
-#### 3.2 Configurations for Windows
+### 3.2 Configurations for Windows
For Intel iGPU
@@ -99,9 +99,9 @@ set SYCL_CACHE_PERSISTENT=1
> [!NOTE]
> For the first time that each model runs on Intel iGPU/Intel Arc™ A300-Series or Pro A60, it may take several minutes to compile.
-### 4. Running examples
+## 4. Running examples
-#### Example 1: Predict Tokens using `generate()` API
+### Example 1: Predict Tokens using `generate()` API
In the example [generate.py](./generate.py), we show a basic use case for a ChatGLM3 model to predict the next N tokens using `generate()` API, with IPEX-LLM INT4 optimizations on Intel GPUs.
```
@@ -141,7 +141,7 @@ What is AI?
AI stands for Artificial Intelligence. It refers to the development of computer systems or machines that can perform tasks that would normally require human intelligence, such as recognizing patterns
```
-#### Example 2: Stream Chat using `stream_chat()` API
+### Example 2: Stream Chat using `stream_chat()` API
In the example [streamchat.py](./streamchat.py), we show a basic use case for a ChatGLM3 model to stream chat, with IPEX-LLM INT4 optimizations.
**Stream Chat using `stream_chat()` API**:
diff --git a/python/llm/example/GPU/HF-Transformers-AutoModels/Model/glm4/README.md b/python/llm/example/GPU/HF-Transformers-AutoModels/Model/glm4/README.md
index 9ee248ca6c5..930e77af04b 100644
--- a/python/llm/example/GPU/HF-Transformers-AutoModels/Model/glm4/README.md
+++ b/python/llm/example/GPU/HF-Transformers-AutoModels/Model/glm4/README.md
@@ -1,11 +1,11 @@
# GLM-4
In this directory, you will find examples on how you could apply IPEX-LLM INT4 optimizations on GLM-4 models on [Intel GPUs](../../../README.md). For illustration purposes, we utilize the [THUDM/glm-4-9b-chat](https://huggingface.co/THUDM/glm-4-9b-chat) as a reference InternLM model.
-### 0. Requirements
+## 0. Requirements
To run these examples with IPEX-LLM on Intel GPUs, we have some recommended requirements for your machine, please refer to [here](../../../README.md#requirements) for more information.
-### 1. Install
-#### 1.1 Installation on Linux
+## 1. Install
+### 1.1 Installation on Linux
We suggest using conda to manage environment:
```bash
conda create -n llm python=3.11
@@ -17,7 +17,7 @@ pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-exte
pip install tiktoken
```
-#### 1.2 Installation on Windows
+### 1.2 Installation on Windows
We suggest using conda to manage environment:
```bash
conda create -n llm python=3.11 libuv
@@ -30,7 +30,7 @@ pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-exte
pip install tiktoken
```
-### 2. Configures OneAPI environment variables for Linux
+## 2. Configures OneAPI environment variables for Linux
> [!NOTE]
> Skip this step if you are running on Windows.
@@ -41,9 +41,9 @@ This is a required step on Linux for APT or offline installed oneAPI. Skip this
source /opt/intel/oneapi/setvars.sh
```
-### 3. Runtime Configurations
+## 3. Runtime Configurations
For optimal performance, it is recommended to set several environment variables. Please check out the suggestions based on your device.
-#### 3.1 Configurations for Linux
+### 3.1 Configurations for Linux
For Intel Arc™ A-Series Graphics and Intel Data Center GPU Flex Series
@@ -80,7 +80,7 @@ export BIGDL_LLM_XMX_DISABLED=1
-#### 3.2 Configurations for Windows
+### 3.2 Configurations for Windows
For Intel iGPU
@@ -104,9 +104,9 @@ set SYCL_CACHE_PERSISTENT=1
> [!NOTE]
> For the first time that each model runs on Intel iGPU/Intel Arc™ A300-Series or Pro A60, it may take several minutes to compile.
-### 4. Running examples
+## 4. Running examples
-#### Example 1: Predict Tokens using `generate()` API
+### Example 1: Predict Tokens using `generate()` API
In the example [generate.py](./generate.py), we show a basic use case for a GLM-4 model to predict the next N tokens using `generate()` API, with IPEX-LLM INT4 optimizations on Intel GPUs.
```
@@ -147,7 +147,7 @@ What is AI?
Artificial Intelligence (AI) refers to the simulation of human intelligence in machines that are programmed to think like humans and mimic their actions. The term "art
```
-#### Example 2: Stream Chat using `stream_chat()` API
+### Example 2: Stream Chat using `stream_chat()` API
In the example [streamchat.py](./streamchat.py), we show a basic use case for a GLM-4 model to stream chat, with IPEX-LLM INT4 optimizations.
**Stream Chat using `stream_chat()` API**:
diff --git a/python/llm/example/GPU/PyTorch-Models/Model/chatglm2/README.md b/python/llm/example/GPU/PyTorch-Models/Model/chatglm2/README.md
index 43e6149df8a..070f754720e 100644
--- a/python/llm/example/GPU/PyTorch-Models/Model/chatglm2/README.md
+++ b/python/llm/example/GPU/PyTorch-Models/Model/chatglm2/README.md
@@ -4,8 +4,8 @@ In this directory, you will find examples on how you could use IPEX-LLM `optimiz
## Requirements
To run these examples with IPEX-LLM on Intel GPUs, we have some recommended requirements for your machine, please refer to [here](../../../README.md#requirements) for more information.
-### 1. Install
-#### 1.1 Installation on Linux
+## 1. Install
+### 1.1 Installation on Linux
We suggest using conda to manage environment:
```bash
conda create -n llm python=3.11
@@ -14,7 +14,7 @@ conda activate llm
pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/
```
-#### 1.2 Installation on Windows
+### 1.2 Installation on Windows
We suggest using conda to manage environment:
```bash
conda create -n llm python=3.11 libuv
@@ -24,7 +24,7 @@ conda activate llm
pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/
```
-### 2. Configures OneAPI environment variables for Linux
+## 2. Configures OneAPI environment variables for Linux
> [!NOTE]
> Skip this step if you are running on Windows.
@@ -35,9 +35,9 @@ This is a required step on Linux for APT or offline installed oneAPI. Skip this
source /opt/intel/oneapi/setvars.sh
```
-### 3. Runtime Configurations
+## 3. Runtime Configurations
For optimal performance, it is recommended to set several environment variables. Please check out the suggestions based on your device.
-#### 3.1 Configurations for Linux
+### 3.1 Configurations for Linux
For Intel Arc™ A-Series Graphics and Intel Data Center GPU Flex Series
@@ -74,7 +74,7 @@ export BIGDL_LLM_XMX_DISABLED=1
-#### 3.2 Configurations for Windows
+### 3.2 Configurations for Windows
For Intel iGPU
@@ -99,9 +99,9 @@ set SYCL_CACHE_PERSISTENT=1
> [!NOTE]
> For the first time that each model runs on Intel iGPU/Intel Arc™ A300-Series or Pro A60, it may take several minutes to compile.
-### 4. Running examples
+## 4. Running examples
-#### Example 1: Predict Tokens using `generate()` API
+### Example 1: Predict Tokens using `generate()` API
In the example [generate.py](./generate.py), we show a basic use case for a ChatGLM2 model to predict the next N tokens using `generate()` API, with IPEX-LLM INT4 optimizations on Intel GPUs.
```bash
@@ -136,7 +136,7 @@ Inference time: xxxx s
答： Artificial Intelligence (AI) refers to the ability of a computer or machine to perform tasks that typically require human-like intelligence, such as understanding language, recognizing patterns
```
-#### Example 2: Stream Chat using `stream_chat()` API
+### Example 2: Stream Chat using `stream_chat()` API
In the example [streamchat.py](./streamchat.py), we show a basic use case for a ChatGLM2 model to stream chat, with IPEX-LLM INT4 optimizations.
**Stream Chat using `stream_chat()` API**:
diff --git a/python/llm/example/GPU/PyTorch-Models/Model/chatglm3/README.md b/python/llm/example/GPU/PyTorch-Models/Model/chatglm3/README.md
index fafe7aac795..335c8ea0762 100644
--- a/python/llm/example/GPU/PyTorch-Models/Model/chatglm3/README.md
+++ b/python/llm/example/GPU/PyTorch-Models/Model/chatglm3/README.md
@@ -4,8 +4,8 @@ In this directory, you will find examples on how you could use IPEX-LLM `optimiz
## Requirements
To run these examples with IPEX-LLM on Intel GPUs, we have some recommended requirements for your machine, please refer to [here](../../../README.md#requirements) for more information.
-### 1. Install
-#### 1.1 Installation on Linux
+## 1. Install
+### 1.1 Installation on Linux
We suggest using conda to manage environment:
```bash
conda create -n llm python=3.11
@@ -14,7 +14,7 @@ conda activate llm
pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/
```
-#### 1.2 Installation on Windows
+### 1.2 Installation on Windows
We suggest using conda to manage environment:
```bash
conda create -n llm python=3.11 libuv
@@ -24,7 +24,7 @@ conda activate llm
pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/
```
-### 2. Configures OneAPI environment variables for Linux
+## 2. Configures OneAPI environment variables for Linux
> [!NOTE]
> Skip this step if you are running on Windows.
@@ -35,9 +35,9 @@ This is a required step on Linux for APT or offline installed oneAPI. Skip this
source /opt/intel/oneapi/setvars.sh
```
-### 3. Runtime Configurations
+## 3. Runtime Configurations
For optimal performance, it is recommended to set several environment variables. Please check out the suggestions based on your device.
-#### 3.1 Configurations for Linux
+### 3.1 Configurations for Linux
For Intel Arc™ A-Series Graphics and Intel Data Center GPU Flex Series
@@ -74,7 +74,7 @@ export BIGDL_LLM_XMX_DISABLED=1
-#### 3.2 Configurations for Windows
+### 3.2 Configurations for Windows
For Intel iGPU
@@ -98,9 +98,9 @@ set SYCL_CACHE_PERSISTENT=1
> [!NOTE]
> For the first time that each model runs on Intel iGPU/Intel Arc™ A300-Series or Pro A60, it may take several minutes to compile.
-### 4. Running examples
+## 4. Running examples
-#### Example 1: Predict Tokens using `generate()` API
+### Example 1: Predict Tokens using `generate()` API
In the example [generate.py](./generate.py), we show a basic use case for a ChatGLM3 model to predict the next N tokens using `generate()` API, with IPEX-LLM INT4 optimizations on Intel GPUs.
```bash
@@ -133,7 +133,7 @@ What is AI?
AI stands for Artificial Intelligence. It refers to the development of computer systems or machines that can perform tasks that would normally require human intelligence, such as recognizing patterns
```
-#### Example 2: Stream Chat using `stream_chat()` API
+### Example 2: Stream Chat using `stream_chat()` API
In the example [streamchat.py](./streamchat.py), we show a basic use case for a ChatGLM3 model to stream chat, with IPEX-LLM INT4 optimizations.
**Stream Chat using `stream_chat()` API**:
From 81a69240b316faf58b4825f9a441f76794800a20 Mon Sep 17 00:00:00 2001
From: jinbridge <2635480475@qq.com>
Date: Thu, 13 Jun 2024 17:05:23 +0800
Subject: [PATCH 3/4] fix section
---
.../GPU/HF-Transformers-AutoModels/Model/chatglm2/README.md | 5 ++---
.../GPU/HF-Transformers-AutoModels/Model/chatglm3/README.md | 5 ++---
.../GPU/HF-Transformers-AutoModels/Model/glm4/README.md | 5 ++---
.../llm/example/GPU/PyTorch-Models/Model/chatglm2/README.md | 5 ++---
.../llm/example/GPU/PyTorch-Models/Model/chatglm3/README.md | 5 ++---
5 files changed, 10 insertions(+), 15 deletions(-)
diff --git a/python/llm/example/GPU/HF-Transformers-AutoModels/Model/chatglm2/README.md b/python/llm/example/GPU/HF-Transformers-AutoModels/Model/chatglm2/README.md
index 9f6a268a30e..407534acbc5 100644
--- a/python/llm/example/GPU/HF-Transformers-AutoModels/Model/chatglm2/README.md
+++ b/python/llm/example/GPU/HF-Transformers-AutoModels/Model/chatglm2/README.md
@@ -114,9 +114,8 @@ Arguments info:
- `--prompt PROMPT`: argument defining the prompt to be infered (with integrated prompt format for chat). It is default to be `'AI是什么？'`.
- `--n-predict N_PREDICT`: argument defining the max number of tokens to predict. It is default to be `32`.
-Sample Output:
-
-[THUDM/chatglm2-6b](https://huggingface.co/THUDM/chatglm2-6b)
+#### Sample Output
+#### [THUDM/chatglm2-6b](https://huggingface.co/THUDM/chatglm2-6b)
```log
Inference time: xxxx s
-------------------- Prompt --------------------
diff --git a/python/llm/example/GPU/HF-Transformers-AutoModels/Model/chatglm3/README.md b/python/llm/example/GPU/HF-Transformers-AutoModels/Model/chatglm3/README.md
index 490da17f0d5..70b66b022a0 100644
--- a/python/llm/example/GPU/HF-Transformers-AutoModels/Model/chatglm3/README.md
+++ b/python/llm/example/GPU/HF-Transformers-AutoModels/Model/chatglm3/README.md
@@ -113,9 +113,8 @@ Arguments info:
- `--prompt PROMPT`: argument defining the prompt to be infered (with integrated prompt format for chat). It is default to be `'AI是什么？'`.
- `--n-predict N_PREDICT`: argument defining the max number of tokens to predict. It is default to be `32`.
-Sample Output:
-
-[THUDM/chatglm3-6b](https://huggingface.co/THUDM/chatglm3-6b)
+#### Sample Output
+#### [THUDM/chatglm3-6b](https://huggingface.co/THUDM/chatglm3-6b)
```log
Inference time: xxxx s
-------------------- Prompt --------------------
diff --git a/python/llm/example/GPU/HF-Transformers-AutoModels/Model/glm4/README.md b/python/llm/example/GPU/HF-Transformers-AutoModels/Model/glm4/README.md
index 930e77af04b..95419e1e2f2 100644
--- a/python/llm/example/GPU/HF-Transformers-AutoModels/Model/glm4/README.md
+++ b/python/llm/example/GPU/HF-Transformers-AutoModels/Model/glm4/README.md
@@ -118,9 +118,8 @@ Arguments info:
- `--prompt PROMPT`: argument defining the prompt to be infered (with integrated prompt format for chat). It is default to be `'AI是什么？'`.
- `--n-predict N_PREDICT`: argument defining the max number of tokens to predict. It is default to be `32`.
-Sample Output:
-
-[THUDM/glm-4-9b-chat](https://huggingface.co/THUDM/glm-4-9b-chat)
+#### Sample Output
+#### [THUDM/glm-4-9b-chat](https://huggingface.co/THUDM/glm-4-9b-chat)
```log
Inference time: xxxx s
-------------------- Prompt --------------------
diff --git a/python/llm/example/GPU/PyTorch-Models/Model/chatglm2/README.md b/python/llm/example/GPU/PyTorch-Models/Model/chatglm2/README.md
index 070f754720e..f3a10f37a6b 100644
--- a/python/llm/example/GPU/PyTorch-Models/Model/chatglm2/README.md
+++ b/python/llm/example/GPU/PyTorch-Models/Model/chatglm2/README.md
@@ -114,9 +114,8 @@ In the example, several arguments can be passed to satisfy your requirements:
- `--prompt PROMPT`: argument defining the prompt to be infered (with integrated prompt format for chat). It is default to be `'AI是什么？'`.
- `--n-predict N_PREDICT`: argument defining the max number of tokens to predict. It is default to be `32`.
-Sample Output:
-
-[THUDM/chatglm2-6b](https://huggingface.co/THUDM/chatglm2-6b)
+#### Sample Output
+#### [THUDM/chatglm2-6b](https://huggingface.co/THUDM/chatglm2-6b)
```log
Inference time: xxxx s
diff --git a/python/llm/example/GPU/PyTorch-Models/Model/chatglm3/README.md b/python/llm/example/GPU/PyTorch-Models/Model/chatglm3/README.md
index 335c8ea0762..6c198bfd5dd 100644
--- a/python/llm/example/GPU/PyTorch-Models/Model/chatglm3/README.md
+++ b/python/llm/example/GPU/PyTorch-Models/Model/chatglm3/README.md
@@ -113,9 +113,8 @@ In the example, several arguments can be passed to satisfy your requirements:
- `--prompt PROMPT`: argument defining the prompt to be infered (with integrated prompt format for chat). It is default to be `'AI是什么？'`.
- `--n-predict N_PREDICT`: argument defining the max number of tokens to predict. It is default to be `32`.
-Sample Output:
-
-[THUDM/chatglm3-6b](https://huggingface.co/THUDM/chatglm3-6b)
+#### Sample Output
+#### [THUDM/chatglm3-6b](https://huggingface.co/THUDM/chatglm3-6b)
```log
Inference time: xxxx s
-------------------- Output --------------------
From 0832c7f55089287963115890314176a3ee0bb30e Mon Sep 17 00:00:00 2001
From: jinbridge <2635480475@qq.com>
Date: Fri, 14 Jun 2024 11:04:44 +0800
Subject: [PATCH 4/4] fix tiktoken
---
.../CPU/HF-Transformers-AutoModels/Model/glm4/README.md | 4 ++--
python/llm/example/CPU/PyTorch-Models/Model/glm4/README.md | 4 ++--
.../GPU/HF-Transformers-AutoModels/Model/glm4/README.md | 4 ++--
python/llm/example/GPU/PyTorch-Models/Model/glm4/README.md | 4 ++--
4 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/python/llm/example/CPU/HF-Transformers-AutoModels/Model/glm4/README.md b/python/llm/example/CPU/HF-Transformers-AutoModels/Model/glm4/README.md
index db2a0c524bb..cf4c99d1201 100644
--- a/python/llm/example/CPU/HF-Transformers-AutoModels/Model/glm4/README.md
+++ b/python/llm/example/CPU/HF-Transformers-AutoModels/Model/glm4/README.md
@@ -18,7 +18,7 @@ conda activate llm
pip install --pre --upgrade ipex-llm[all] --extra-index-url https://download.pytorch.org/whl/cpu
# install tiktoken required for GLM-4
-pip install tiktoken
+pip install "tiktoken>=0.7.0"
```
On Windows:
@@ -29,7 +29,7 @@ conda activate llm
pip install --pre --upgrade ipex-llm[all]
-pip install tiktoken
+pip install "tiktoken>=0.7.0"
```
## 2. Run
diff --git a/python/llm/example/CPU/PyTorch-Models/Model/glm4/README.md b/python/llm/example/CPU/PyTorch-Models/Model/glm4/README.md
index 9a1cf56069d..2af60c6f49a 100644
--- a/python/llm/example/CPU/PyTorch-Models/Model/glm4/README.md
+++ b/python/llm/example/CPU/PyTorch-Models/Model/glm4/README.md
@@ -21,7 +21,7 @@ conda activate llm
pip install --pre --upgrade ipex-llm[all] --extra-index-url https://download.pytorch.org/whl/cpu
# install tiktoken required for GLM-4
-pip install tiktoken
+pip install "tiktoken>=0.7.0"
```
On Windows:
@@ -32,7 +32,7 @@ conda activate llm
pip install --pre --upgrade ipex-llm[all]
-pip install tiktoken
+pip install "tiktoken>=0.7.0"
```
### 2. Run
diff --git a/python/llm/example/GPU/HF-Transformers-AutoModels/Model/glm4/README.md b/python/llm/example/GPU/HF-Transformers-AutoModels/Model/glm4/README.md
index 95419e1e2f2..14d3f645f61 100644
--- a/python/llm/example/GPU/HF-Transformers-AutoModels/Model/glm4/README.md
+++ b/python/llm/example/GPU/HF-Transformers-AutoModels/Model/glm4/README.md
@@ -14,7 +14,7 @@ conda activate llm
pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/
# install tiktoken required for GLM-4
-pip install tiktoken
+pip install "tiktoken>=0.7.0"
```
### 1.2 Installation on Windows
@@ -27,7 +27,7 @@ conda activate llm
pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/
# install tiktoken required for GLM-4
-pip install tiktoken
+pip install "tiktoken>=0.7.0"
```
## 2. Configures OneAPI environment variables for Linux
diff --git a/python/llm/example/GPU/PyTorch-Models/Model/glm4/README.md b/python/llm/example/GPU/PyTorch-Models/Model/glm4/README.md
index 6993df68438..f7e95fddacc 100644
--- a/python/llm/example/GPU/PyTorch-Models/Model/glm4/README.md
+++ b/python/llm/example/GPU/PyTorch-Models/Model/glm4/README.md
@@ -16,7 +16,7 @@ conda activate llm
pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/
# install tiktoken required for GLM-4
-pip install tiktoken
+pip install "tiktoken>=0.7.0"
```
#### 1.2 Installation on Windows
@@ -29,7 +29,7 @@ conda activate llm
pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/
# install tiktoken required for GLM-4
-pip install tiktoken
+pip install "tiktoken>=0.7.0"
```
### 2. Configures OneAPI environment variables for Linux