modify

python/llm/example/CPU/StableDiffusion/README.md

@@ -2,23 +2,13 @@
 In this directory, you will find examples on how to run StableDiffusion models on CPU.
 
 ### 1. Installation
-#### 1.1 Installation on Linux
-We suggest using conda to manage environment. 
-```bash
-conda create -n diffusion python=3.11
-conda activate diffusion
-pip install --pre --upgrade ipex-llm[all] --extra-index-url https://download.pytorch.org/whl/cpu
-pip install diffusers["torch"] transformers
-pip install -U PEFT transformers
-pip install setuptools==69.5.1
-```
+#### 1.1. Install IPEX-LLM
+Follow the instructions in [IPEX-LLM CPU installation guide](https://ipex-llm.readthedocs.io/en/latest/doc/LLM/Overview/install_cpu.html) to install ipex-llm. We recommend to use miniconda to manage your python environment.
 
-#### 1.2 Installation on Windows
-We suggest using conda to manage environment. 
+#### 1.2 Install dependencies for Stable Diffusion
+Assume you have created a conda environment named diffusion with ipex-llm installed. Run below commands to install dependencies for running Stable Diffusion.
 ```bash
-conda create -n diffusion python=3.11 libuv
 conda activate diffusion
-pip install --pre --upgrade ipex-llm[all] --extra-index-url https://download.pytorch.org/whl/cpu
 pip install diffusers["torch"] transformers
 pip install -U PEFT transformers
 pip install setuptools==69.5.1
@@ -38,3 +28,15 @@ Arguments info:
 - `--save-path`: argument defining the path to save the generated figure. It is default to be `sdxl-cpu.png`.
 - `--num-steps`: argument defining the number of inference steps. It is default to be `20`. 
 
+#### 4.2 LCM-LoRA Example
+The example shows how to performing inference with LCM-LoRA on Intel CPU.
+```bash
+python ./lora-lcm.py
+```
+
+Arguments info:
+- `--repo-id-or-model-path REPO_ID_OR_MODEL_PATH`: argument defining the huggingface repo id for the stable diffusion xl model (e.g. `stabilityai/stable-diffusion-xl-base-1.0`) to be downloaded, or the path to the huggingface checkpoint folder. It is default to be `'stabilityai/stable-diffusion-xl-base-1.0'`.
+- `--lora-weights-path`: argument defining the huggingface repo id for the LCM-LoRA model (e.g. `latent-consistency/lcm-lora-sdxl`) to be downloaded, or the path to huggingface checkpoint folder. It is default to be `'latent-consistency/lcm-lora-sdxl'`. 
+- `--prompt PROMPT`: argument defining the prompt to be infered. It is default to be `'A lovely dog on the table, detailed, 8k'`.
+- `--save-path`: argument defining the path to save the generated figure. It is default to be `lcm-lora-sdxl-cpu.png`.
+- `--num-steps`: argument defining the number of inference steps. It is default to be `4`.

python/llm/example/CPU/StableDiffusion/lora-lcm.py

@@ -0,0 +1,55 @@
+#
+# Copyright 2016 The BigDL Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Code is adapted from https://huggingface.co/docs/diffusers/main/en/using-diffusers/inference_with_lcm_lora
+
+import torch
+from diffusers import DiffusionPipeline, LCMScheduler
+import ipex_llm
+import argparse
+
+
+def main(args):
+    pipe = DiffusionPipeline.from_pretrained(
+        args.repo_id_or_model_path,
+        torch_dtype=torch.bfloat16,
+    ).to("cpu")
+
+    # set scheduler
+    pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config)
+
+    # load LCM-LoRA
+    pipe.load_lora_weights(args.lora_weights_path)
+
+    generator = torch.manual_seed(42)
+    image = pipe(
+        prompt=args.prompt, num_inference_steps=args.num_steps, generator=generator, guidance_scale=1.0
+    ).images[0]
+    image.save(args.save_path)
+
+if __name__=="__main__":
+    parser = argparse.ArgumentParser(description="Stable Diffusion lora-lcm")
+    parser.add_argument('--repo-id-or-model-path', type=str, default="stabilityai/stable-diffusion-xl-base-1.0",
+                        help='The huggingface repo id for the stable diffusion model checkpoint')
+    parser.add_argument('--lora-weights-path',type=str,default="latent-consistency/lcm-lora-sdxl",
+                        help='The huggingface repo id for the lcm lora sdxl checkpoint')
+    parser.add_argument('--prompt', type=str, default="A lovely dog on the table, detailed, 8k",
+                        help='Prompt to infer')
+    parser.add_argument('--save-path',type=str,default="lcm-lora-sdxl-cpu.png",
+                        help="Path to save the generated figure")
+    parser.add_argument('--num-steps',type=int,default=4,
+                        help="Number of inference steps")
+    args = parser.parse_args()
+    main(args)

python/llm/example/CPU/StableDiffusion/sdxl.py

@@ -13,6 +13,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 #
+# Code is adapted from https://huggingface.co/docs/diffusers/en/using-diffusers/sdxl
 
 from diffusers import AutoPipelineForText2Image
 import torch

python/llm/example/GPU/StableDiffusion/README.md

@@ -2,22 +2,13 @@
 In this directory, you will find examples on how to run StableDiffusion models on [Intel GPUs](../README.md).
 
 ### 1. Installation
-#### 1.1 Installation on Linux
-We suggest using conda to manage environment. 
-```bash
-conda create -n diffusion python=3.11
-conda activate diffusion
-pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/
-pip install diffusers["torch"] transformers
-pip install -U PEFT transformers
-```
+#### 1.1 Install IPEX-LLM
+Follow the instructions in IPEX-GPU installation guides ([Linux Guide](https://ipex-llm.readthedocs.io/en/latest/doc/LLM/Quickstart/install_linux_gpu.html), [Windows Guide](https://ipex-llm.readthedocs.io/en/latest/doc/LLM/Quickstart/install_windows_gpu.html)) according to your system to install IPEX-LLM. After the installation, you should have created a conda environment, named diffusion for instance. 
 
-#### 1.2 Installation on Windows
-We suggest using conda to manage environment. 
+#### 1.2 Install dependencies for Stable Diffusion
+Assume you have created a conda environment named diffusion with ipex-llm installed. Run below commands to install dependencies for running Stable Diffusion.
 ```bash
-conda create -n diffusion python=3.11 libuv
 conda activate diffusion
-pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/
 pip install diffusers["torch"] transformers
 pip install -U PEFT transformers
 ```

python/llm/example/GPU/StableDiffusion/lora-lcm.py

@@ -13,6 +13,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 #
+# Code is adapted from https://huggingface.co/docs/diffusers/main/en/using-diffusers/inference_with_lcm_lora
 
 import torch
 from diffusers import DiffusionPipeline, LCMScheduler

python/llm/example/GPU/StableDiffusion/sdxl.py

@@ -13,6 +13,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 #
+# Code is adapted from https://huggingface.co/docs/diffusers/en/using-diffusers/sdxl
 
 from diffusers import AutoPipelineForText2Image
 import torch