Skip to content

Commit

Permalink
small fix of cpp quickstart(#10829)
Browse files Browse the repository at this point in the history
  • Loading branch information
rnwang04 authored Apr 22, 2024
1 parent 61c67af commit 1edb19c
Show file tree
Hide file tree
Showing 3 changed files with 41 additions and 3 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ This quickstart guide walks you through how to run Llama 3 on Intel GPU using `l

#### 1.1 Install IPEX-LLM for llama.cpp and Initialize

Visit [Run llama.cpp with IPEX-LLM on Intel GPU Guide](https://ipex-llm.readthedocs.io/en/latest/doc/LLM/Quickstart/llama_cpp_quickstart.html), and follow the instructions in section [Prerequisites](https://ipex-llm.readthedocs.io/en/latest/doc/LLM/Quickstart/llama_cpp_quickstart.html#prerequisites) to setup and section [Install IPEX-LLM for llama.cpp](https://ipex-llm.readthedocs.io/en/latest/doc/LLM/Quickstart/llama_cpp_quickstart.html#install-ipex-llm-for-llama-cpp) to install the IPEX-LLM with llama.cpp binaries, then follow the instructions in section [Initialize llama.cpp with IPEX-LLM](https://ipex-llm.readthedocs.io/en/latest/doc/LLM/Quickstart/llama_cpp_quickstart.html#prerequisites) to initialize.
Visit [Run llama.cpp with IPEX-LLM on Intel GPU Guide](https://ipex-llm.readthedocs.io/en/latest/doc/LLM/Quickstart/llama_cpp_quickstart.html), and follow the instructions in section [Prerequisites](https://ipex-llm.readthedocs.io/en/latest/doc/LLM/Quickstart/llama_cpp_quickstart.html#prerequisites) to setup and section [Install IPEX-LLM for llama.cpp](https://ipex-llm.readthedocs.io/en/latest/doc/LLM/Quickstart/llama_cpp_quickstart.html#install-ipex-llm-for-llama-cpp) to install the IPEX-LLM with llama.cpp binaries, then follow the instructions in section [Initialize llama.cpp with IPEX-LLM](https://ipex-llm.readthedocs.io/en/latest/doc/LLM/Quickstart/llama_cpp_quickstart.html#initialize-llama-cpp-with-ipex-llm) to initialize.

**After above steps, you should have created a conda environment, named `llm-cpp` for instance and have llama.cpp binaries in your current directory.**

Expand All @@ -29,6 +29,33 @@ Suppose you have downloaded a [Meta-Llama-3-8B-Instruct-Q4_K_M.gguf](https://hug

#### 1.3 Run Llama3 on Intel GPU using llama.cpp

##### Set Environment Variables(optional)

```eval_rst
.. note::
This is a required step on for APT or offline installed oneAPI. Skip this step for PIP-installed oneAPI.
```

Configure oneAPI variables by running the following command:

```eval_rst
.. tabs::
.. tab:: Linux
.. code-block:: bash
source /opt/intel/oneapi/setvars.sh
.. tab:: Windows
.. code-block:: bash
call "C:\Program Files (x86)\Intel\oneAPI\setvars.bat"
```

##### Run llama3

Under your current directory, exceuting below command to do inference with Llama3:

```eval_rst
Expand Down Expand Up @@ -99,6 +126,7 @@ Launch the Ollama service:
export ZES_ENABLE_SYSMAN=1
export SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=1
export OLLAMA_NUM_GPU=999
# Below is a required step for APT or offline installed oneAPI. Skip below step for PIP-installed oneAPI.
source /opt/intel/oneapi/setvars.sh
./ollama serve
Expand All @@ -112,6 +140,7 @@ Launch the Ollama service:
set no_proxy=localhost,127.0.0.1
set ZES_ENABLE_SYSMAN=1
set OLLAMA_NUM_GPU=999
# Below is a required step for APT or offline installed oneAPI. Skip below step for PIP-installed oneAPI.
call "C:\Program Files (x86)\Intel\oneAPI\setvars.bat"
ollama serve
Expand All @@ -124,7 +153,7 @@ Launch the Ollama service:
To allow the service to accept connections from all IP addresses, use `OLLAMA_HOST=0.0.0.0 ./ollama serve` instead of just `./ollama serve`.
```

#### 2.2.2 Using Ollama Run Llama3
##### 2.2.2 Using Ollama Run Llama3

Keep the Ollama service on and open another terminal and run llama3 with `ollama run`:

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,14 @@ Then you can use following command to initialize `llama.cpp` with IPEX-LLM:

Here we provide a simple example to show how to run a community GGUF model with IPEX-LLM.

#### Set Environment Variables
#### Set Environment Variables(optional)

```eval_rst
.. note::
This is a required step on for APT or offline installed oneAPI. Skip this step for PIP-installed oneAPI.
```

Configure oneAPI variables by running the following command:

```eval_rst
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@ You may launch the Ollama service as below:
export OLLAMA_NUM_GPU=999
export no_proxy=localhost,127.0.0.1
export ZES_ENABLE_SYSMAN=1
# Below is a required step for APT or offline installed oneAPI. Skip below step for PIP-installed oneAPI.
source /opt/intel/oneapi/setvars.sh
./ollama serve
Expand All @@ -68,6 +69,7 @@ You may launch the Ollama service as below:
set OLLAMA_NUM_GPU=999
set no_proxy=localhost,127.0.0.1
set ZES_ENABLE_SYSMAN=1
# Below is a required step for APT or offline installed oneAPI. Skip below step for PIP-installed oneAPI.
call "C:\Program Files (x86)\Intel\oneAPI\setvars.bat"
ollama serve
Expand Down

0 comments on commit 1edb19c

Please sign in to comment.