Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Stable Diffusion examples on GPU and CPU #11166

Merged
merged 9 commits into from
Jun 12, 2024
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion python/llm/example/CPU/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ This folder contains examples of running IPEX-LLM on Intel CPU:
- [Native-Models](Native-Models): converting & running LLM in `llama`/`chatglm`/`bloom`/`gptneox`/`starcoder` model family using native (cpp) implementation
- [Speculative-Decoding](Speculative-Decoding): running any ***Hugging Face Transformers*** model with ***self-speculative decoding*** on Intel CPUs
- [ModelScope-Models](ModelScope-Models): running ***ModelScope*** model with IPEX-LLM on Intel CPUs

- [StableDiffusion-Models](StableDiffusion): running **stable diffusion** models on Intel CPUs.

## System Support
**Hardware**:
Expand Down
40 changes: 40 additions & 0 deletions python/llm/example/CPU/StableDiffusion/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# Stable Diffusion
In this directory, you will find examples on how to run StableDiffusion models on CPU.

### 1. Installation
#### 1.1 Installation on Linux
ivy-lv11 marked this conversation as resolved.
Show resolved Hide resolved
We suggest using conda to manage environment.
```bash
conda create -n diffusion python=3.11
conda activate diffusion
pip install --pre --upgrade ipex-llm[all] --extra-index-url https://download.pytorch.org/whl/cpu
pip install diffusers["torch"] transformers
pip install -U PEFT transformers
pip install setuptools==69.5.1
```

#### 1.2 Installation on Windows
We suggest using conda to manage environment.
```bash
conda create -n diffusion python=3.11 libuv
conda activate diffusion
pip install --pre --upgrade ipex-llm[all] --extra-index-url https://download.pytorch.org/whl/cpu
pip install diffusers["torch"] transformers
pip install -U PEFT transformers
pip install setuptools==69.5.1
```

### 2. Examples

#### 2.1 StableDiffusion XL Example
The example shows how to run StableDiffusion XL example on Intel CPU.
```bash
python ./sdxl.py
```

Arguments info:
- `--repo-id-or-model-path REPO_ID_OR_MODEL_PATH`: argument defining the huggingface repo id for the stable diffusion xl model (e.g. `stabilityai/stable-diffusion-xl-base-1.0`) to be downloaded, or the path to the huggingface checkpoint folder. It is default to be `'stabilityai/stable-diffusion-xl-base-1.0'`.
- `--prompt PROMPT`: argument defining the prompt to be infered. It is default to be `'A lovely dog on the table, detailed, 8k'`.
- `--save-path`: argument defining the path to save the generated figure. It is default to be `sdxl-cpu.png`.
- `--num-steps`: argument defining the number of inference steps. It is default to be `20`.

46 changes: 46 additions & 0 deletions python/llm/example/CPU/StableDiffusion/sdxl.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
#
# Copyright 2016 The BigDL Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

from diffusers import AutoPipelineForText2Image
import torch
import ipex_llm
import numpy as np
from PIL import Image
import argparse


def main(args):
pipeline_text2image = AutoPipelineForText2Image.from_pretrained(
args.repo_id_or_model_path,
torch_dtype=torch.float16,
use_safetensors=True
).to("cpu")

image = pipeline_text2image(prompt=args.prompt,num_inference_steps=args.num_steps).images[0]
image.save(args.save_path)

if __name__=="__main__":
parser = argparse.ArgumentParser(description="Stable Diffusion")
parser.add_argument('--repo-id-or-model-path', type=str, default="stabilityai/stable-diffusion-xl-base-1.0",
help='The huggingface repo id for the stable diffusion model checkpoint')
parser.add_argument('--prompt', type=str, default="A lovely dog on the table, detailed, 8k",
help='Prompt to infer')
parser.add_argument('--save-path',type=str,default="sdxl-cpu.png",
help="Path to save the generated figure")
parser.add_argument('--num-steps',type=int,default=20,
help="Number of inference steps")
args = parser.parse_args()
main(args)
1 change: 1 addition & 0 deletions python/llm/example/GPU/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ This folder contains examples of running IPEX-LLM on Intel GPU:
- [Speculative-Decoding](Speculative-Decoding): running any ***Hugging Face Transformers*** model with ***self-speculative decoding*** on Intel GPUs
- [ModelScope-Models](ModelScope-Models): running ***ModelScope*** model with IPEX-LLM on Intel GPUs
- [Long-Context](Long-Context): running **long-context** generation with IPEX-LLM on Intel Arc™ A770 Graphics.
- [StableDiffusion](StableDiffusion): running **stable diffusion** with IPEX-LLM on Intel GPUs.


## System Support
Expand Down
124 changes: 124 additions & 0 deletions python/llm/example/GPU/StableDiffusion/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,124 @@
# Stable Diffusion
In this directory, you will find examples on how to run StableDiffusion models on [Intel GPUs](../README.md).

### 1. Installation
#### 1.1 Installation on Linux
We suggest using conda to manage environment.
Copy link
Contributor

@shane-huang shane-huang May 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

separate ipex-llm installation from diffusers dependences. Like below:

1.1. Install IPEX-LLM

Follow the instructions in IPEX-GPU installation guides (Linux Guide, Windows Guide) according to your system to install IPEX-LLM. After the installation, you should have created a conda environment, named diffusion for instance

1.2 Install dependencies for Stable Diffusion

Assume you have created a conda environment named diffusion with ipex-llm installed. Run below commands to install dependencies for running Stable Diffusion.

conda activate diffusion
pip install ...
...

```bash
conda create -n diffusion python=3.11
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove the conda env creation part but use conda activate diffusion instead, as ipex-llm installation part has created it.

conda activate diffusion
pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/
pip install diffusers["torch"] transformers
pip install -U PEFT transformers
```

#### 1.2 Installation on Windows
We suggest using conda to manage environment.
```bash
conda create -n diffusion python=3.11 libuv
conda activate diffusion
pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/
pip install diffusers["torch"] transformers
pip install -U PEFT transformers
```

### 2. Configures OneAPI environment variables for Linux

> [!NOTE]
> Skip this step if you are running on Windows.

This is a required step on Linux for APT or offline installed oneAPI. Skip this step for PIP-installed oneAPI.

```bash
source /opt/intel/oneapi/setvars.sh
```

### 3. Runtime Configurations
For optimal performance, it is recommended to set several environment variables. Please check out the suggestions based on your device.
#### 3.1 Configurations for Linux
<details>

<summary>For Intel Arc™ A-Series Graphics and Intel Data Center GPU Flex Series</summary>

```bash
export USE_XETLA=OFF
export SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=1
export SYCL_CACHE_PERSISTENT=1
```

</details>

<details>

<summary>For Intel Data Center GPU Max Series</summary>

```bash
export LD_PRELOAD=${LD_PRELOAD}:${CONDA_PREFIX}/lib/libtcmalloc.so
export SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=1
export SYCL_CACHE_PERSISTENT=1
export ENABLE_SDP_FUSION=1
```
</details>

<details>

<summary>For Intel iGPU</summary>

```bash
export SYCL_CACHE_PERSISTENT=1
export BIGDL_LLM_XMX_DISABLED=1
```

</details>

#### 3.2 Configurations for Windows
<details>

<summary>For Intel iGPU</summary>

```cmd
set SYCL_CACHE_PERSISTENT=1
set BIGDL_LLM_XMX_DISABLED=1
```

</details>

<details>

<summary>For Intel Arc™ A-Series Graphics</summary>

```cmd
set SYCL_CACHE_PERSISTENT=1
```

</details>

> [!NOTE]
> For the first time that each model runs on Intel iGPU/Intel Arc™ A300-Series or Pro A60, it may take several minutes to compile.

### 4. Examples

#### 4.1 StableDiffusion XL Example
The example shows how to run StableDiffusion XL example on Intel GPU.
```bash
python ./sdxl.py
```

Arguments info:
- `--repo-id-or-model-path REPO_ID_OR_MODEL_PATH`: argument defining the huggingface repo id for the stable diffusion xl model (e.g. `stabilityai/stable-diffusion-xl-base-1.0`) to be downloaded, or the path to the huggingface checkpoint folder. It is default to be `'stabilityai/stable-diffusion-xl-base-1.0'`.
- `--prompt PROMPT`: argument defining the prompt to be infered. It is default to be `'A lovely dog on the table, detailed, 8k'`.
- `--save-path`: argument defining the path to save the generated figure. It is default to be `sdxl-gpu.png`.
- `--num-steps`: argument defining the number of inference steps. It is default to be `20`.

#### 4.2 LCM-LoRA Example
The example shows how to performing inference with LCM-LoRA on Intel GPU.
```bash
python ./lora-lcm.py
```

Arguments info:
- `--repo-id-or-model-path REPO_ID_OR_MODEL_PATH`: argument defining the huggingface repo id for the stable diffusion xl model (e.g. `stabilityai/stable-diffusion-xl-base-1.0`) to be downloaded, or the path to the huggingface checkpoint folder. It is default to be `'stabilityai/stable-diffusion-xl-base-1.0'`.
- `--lora-weights-path`: argument defining the huggingface repo id for the LCM-LoRA model (e.g. `latent-consistency/lcm-lora-sdxl`) to be downloaded, or the path to huggingface checkpoint folder. It is default to be `'latent-consistency/lcm-lora-sdxl'`.
- `--prompt PROMPT`: argument defining the prompt to be infered. It is default to be `'A lovely dog on the table, detailed, 8k'`.
- `--save-path`: argument defining the path to save the generated figure. It is default to be `lcm-lora-sdxl-gpu.png`.
- `--num-steps`: argument defining the number of inference steps. It is default to be `4`.
54 changes: 54 additions & 0 deletions python/llm/example/GPU/StableDiffusion/lora-lcm.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
#
# Copyright 2016 The BigDL Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

ivy-lv11 marked this conversation as resolved.
Show resolved Hide resolved
import torch
from diffusers import DiffusionPipeline, LCMScheduler
import ipex_llm
import argparse


def main(args):
pipe = DiffusionPipeline.from_pretrained(
args.repo_id_or_model_path,
torch_dtype=torch.bfloat16,
).to("xpu")

# set scheduler
pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config)

# load LCM-LoRA
pipe.load_lora_weights(args.lora_weights_path)

generator = torch.manual_seed(42)
image = pipe(
prompt=args.prompt, num_inference_steps=args.num_steps, generator=generator, guidance_scale=1.0
).images[0]
image.save(args.save_path)

if __name__=="__main__":
parser = argparse.ArgumentParser(description="Stable Diffusion lora-lcm")
parser.add_argument('--repo-id-or-model-path', type=str, default="stabilityai/stable-diffusion-xl-base-1.0",
help='The huggingface repo id for the stable diffusion model checkpoint')
parser.add_argument('--lora-weights-path',type=str,default="latent-consistency/lcm-lora-sdxl",
help='The huggingface repo id for the lcm lora sdxl checkpoint')
parser.add_argument('--prompt', type=str, default="A lovely dog on the table, detailed, 8k",
help='Prompt to infer')
parser.add_argument('--save-path',type=str,default="lcm-lora-sdxl-gpu.png",
help="Path to save the generated figure")
parser.add_argument('--num-steps',type=int,default=4,
help="Number of inference steps")
args = parser.parse_args()
main(args)
46 changes: 46 additions & 0 deletions python/llm/example/GPU/StableDiffusion/sdxl.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
#
# Copyright 2016 The BigDL Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

from diffusers import AutoPipelineForText2Image
import torch
import ipex_llm
import numpy as np
from PIL import Image
import argparse


def main(args):
pipeline_text2image = AutoPipelineForText2Image.from_pretrained(
args.repo_id_or_model_path,
torch_dtype=torch.bfloat16,
use_safetensors=True
).to("xpu")

image = pipeline_text2image(prompt=args.prompt,num_inference_steps=args.num_steps).images[0]
image.save(args.save_path)

if __name__=="__main__":
parser = argparse.ArgumentParser(description="Stable Diffusion")
parser.add_argument('--repo-id-or-model-path', type=str, default="stabilityai/stable-diffusion-xl-base-1.0",
help='The huggingface repo id for the stable diffusion model checkpoint')
parser.add_argument('--prompt', type=str, default="A lovely dog on the table, detailed, 8k",
help='Prompt to infer')
parser.add_argument('--save-path',type=str,default="sdxl-gpu.png",
help="Path to save the generated figure")
parser.add_argument('--num-steps',type=int,default=20,
help="Number of inference steps")
args = parser.parse_args()
main(args)
Loading