- News | - Methodology | - Capabilities | - Quick Start | - Finetune | - License | - Citation -
-
-
-
-
-## 1. News
-- 2024-10-28: We release new version of inference code, optimizing the memory usage and time cost. You can refer to [docs/inference.md](docs/inference.md#requiremented-resources) for detailed information.
-- 2024-10-22: :fire: We release the code for OmniGen. Inference: [docs/inference.md](docs/inference.md) Train: [docs/fine-tuning.md](docs/fine-tuning.md)
-- 2024-10-22: :fire: We release the first version of OmniGen. Model Weight: [Shitao/OmniGen-v1](https://huggingface.co/Shitao/OmniGen-v1) HF Demo: [🤗](https://huggingface.co/spaces/Shitao/OmniGen)
-
-
-## 2. Overview
-
-OmniGen is a unified image generation model that can generate a wide range of images from multi-modal prompts. It is designed to be simple, flexible and easy to use. We provide [inference code](#5-quick-start) so that everyone can explore more functionalities of OmniGen.
-
-Existing image generation models often require loading several additional network modules (such as ControlNet, IP-Adapter, Reference-Net, etc.) and performing extra preprocessing steps (e.g., face detection, pose estimation, cropping, etc.) to generate a satisfactory image. However, **we believe that the future image generation paradigm should be more simple and flexible, that is, generating various images directly through arbitrarily multi-modal instructions without the need for additional plugins and operations, similar to how GPT works in language generation.**
-
-Due to the limited resources, OmniGen still has room for improvement. We will continue to optimize it, and hope it inspire more universal image generation models. You can also easily fine-tune OmniGen without worrying about designing networks for specific tasks; you just need to prepare the corresponding data, and then run the [script](#6-finetune). Imagination is no longer limited; everyone can construct any image generation task, and perhaps we can achieve very interesting, wonderful and creative things.
-
-If you have any questions, ideas or interesting tasks you want OmniGen to accomplish, feel free to discuss with us: 2906698981@qq.com, wangyueze@tju.edu.cn, zhengliu1026@gmail.com. We welcome any feedback to help us improve the model.
-
-
-
-
-## 3. Methodology
-
-You can see details in our [paper](https://arxiv.org/abs/2409.11340).
-
-
-## 4. What Can OmniGen do?
-
-
-OmniGen is a unified image generation model that you can use to perform various tasks, including but not limited to text-to-image generation, subject-driven generation, Identity-Preserving Generation, image editing, and image-conditioned generation. **OmniGen don't need additional plugins or operations, it can automatically identify the features (e.g., required object, human pose, depth mapping) in input images according the text prompt.**
-We showcase some examples in [inference.ipynb](inference.ipynb). And in [inference_demo.ipynb](inference_demo.ipynb), we show an interesting pipeline to generate and modify a image.
-
-Here is the illustration of OmniGen's capabilities:
-- You can control the image generation flexibly via OmniGen
-
-- Referring Expression Generation: You can input multiple images and use simple, general language to refer to the objects within those images. OmniGen can automatically recognize the necessary objects in each image and generate new images based on them. No additional operations, such as image cropping or face detection, are required.
-
-
-If you are not entirely satisfied with certain functionalities or wish to add new capabilities, you can try [fine-tuning OmniGen](#6-finetune).
-
-
-
-## 5. Quick Start
-
-
-### Using OmniGen
-Install via Github:
-```bash
-git clone https://github.com/staoxiao/OmniGen.git
-cd OmniGen
-pip install -e .
-```
-
-Here are some examples:
-```python
-from OmniGen import OmniGenPipeline
-
-pipe = OmniGenPipeline.from_pretrained("Shitao/OmniGen-v1")
-
-# Text to Image
-images = pipe(
- prompt="A curly-haired man in a red shirt is drinking tea.",
- height=1024,
- width=1024,
- guidance_scale=2.5,
- seed=0,
-)
-images[0].save("example_t2i.png") # save output PIL Image
-
-# Multi-modal to Image
-# In prompt, we use the placeholder to represent the image. The image placeholder should be in the format of <|image_*|>
-# You can add multiple images in the input_images. Please ensure that each image has its placeholder. For example, for the list input_images [img1_path, img2_path], the prompt needs to have two placeholders:
<|image_1|>,
<|image_2|>.
-images = pipe(
- prompt="A man in a black shirt is reading a book. The man is the right man in
<|image_1|>.",
- input_images=["./imgs/test_cases/two_man.jpg"],
- height=1024,
- width=1024,
- guidance_scale=2.5,
- img_guidance_scale=1.6,
- seed=0
-)
-images[0].save("example_ti2i.png") # save output PIL image
-```
-- If out of memory, you can set `offload_model=True`. If inference time is too long when input multiple images, you can reduce the `max_input_image_size`. For thre required resources and the method to run OmniGen efficiently, please refer to [docs/inference.md#requiremented-resources](docs/inference.md#requiremented-resources).
-- For more examples for image generation, you can refer to [inference.ipynb](inference.ipynb) and [inference_demo.ipynb](inference_demo.ipynb)
-- For more details about the argument in inference, please refer to [docs/inference.md](docs/inference.md).
-
-
-### Using Diffusers
-Coming soon.
-
-
-### Gradio Demo
-
-We construct an online demo in [Huggingface](https://huggingface.co/spaces/Shitao/OmniGen).
-
-For the local gradio demo, you need to install `pip install gradio spaces` , and then you can run:
-```python
-pip install gradio spaces
-python app.py
-```
-
-
-
-## 6. Finetune
-We provide a training script `train.py` to fine-tune OmniGen.
-Here is a toy example about LoRA finetune:
-```bash
-accelerate launch --num_processes=1 train.py \
- --model_name_or_path Shitao/OmniGen-v1 \
- --batch_size_per_device 2 \
- --condition_dropout_prob 0.01 \
- --lr 1e-3 \
- --use_lora \
- --lora_rank 8 \
- --json_file ./toy_data/toy_subject_data.jsonl \
- --image_path ./toy_data/images \
- --max_input_length_limit 18000 \
- --keep_raw_resolution \
- --max_image_size 1024 \
- --gradient_accumulation_steps 1 \
- --ckpt_every 10 \
- --epochs 200 \
- --log_every 1 \
- --results_dir ./results/toy_finetune_lora
-```
-
-Please refer to [docs/fine-tuning.md](docs/fine-tuning.md) for more details (e.g. full finetune).
-
-
-
-## License
-This repo is licensed under the [MIT License](LICENSE).
-
-
-## Citation
-If you find this repository useful, please consider giving a star ⭐ and citation
-```
-@article{xiao2024omnigen,
- title={Omnigen: Unified image generation},
- author={Xiao, Shitao and Wang, Yueze and Zhou, Junjie and Yuan, Huaying and Xing, Xingrun and Yan, Ruiran and Wang, Shuting and Huang, Tiejun and Liu, Zheng},
- journal={arXiv preprint arXiv:2409.11340},
- year={2024}
-}
-```
-
-
-
-
+# OmniGen: Unified Image Generation
+This is the homepage source code of OmniGen, which refers to [LLaVA](https://llava-vl.github.io/).
\ No newline at end of file
diff --git a/app.py b/app.py
deleted file mode 100644
index 0a78e1a..0000000
--- a/app.py
+++ /dev/null
@@ -1,426 +0,0 @@
-import gradio as gr
-from PIL import Image
-import os
-
-import spaces
-
-from OmniGen import OmniGenPipeline
-
-pipe = OmniGenPipeline.from_pretrained(
- "Shitao/OmniGen-v1"
-)
-
-@spaces.GPU(duration=160)
-def generate_image(text, img1, img2, img3, height, width, guidance_scale, img_guidance_scale, inference_steps, seed, separate_cfg_infer, offload_model,
- use_input_image_size_as_output, max_input_image_size):
- input_images = [img1, img2, img3]
- # Delete None
- input_images = [img for img in input_images if img is not None]
- if len(input_images) == 0:
- input_images = None
-
- output = pipe(
- prompt=text,
- input_images=input_images,
- height=height,
- width=width,
- guidance_scale=guidance_scale,
- img_guidance_scale=img_guidance_scale,
- num_inference_steps=inference_steps,
- separate_cfg_infer=separate_cfg_infer,
- use_kv_cache=True,
- offload_kv_cache=True,
- offload_model=offload_model,
- use_input_image_size_as_output=use_input_image_size_as_output,
- seed=seed,
- max_input_image_size=max_input_image_size,
- )
- img = output[0]
- return img
-
-
-
-def get_example():
- case = [
- [
- "A curly-haired man in a red shirt is drinking tea.",
- None,
- None,
- None,
- 1024,
- 1024,
- 2.5,
- 1.6,
- 50,
- 0,
- True,
- False,
- False,
- 1024,
- ],
- [
- "The woman in
<|image_1|> waves her hand happily in the crowd",
- "./imgs/test_cases/zhang.png",
- None,
- None,
- 1024,
- 1024,
- 2.5,
- 1.9,
- 50,
- 128,
- True,
- False,
- False,
- 1024,
- ],
- [
- "A man in a black shirt is reading a book. The man is the right man in
<|image_1|>.",
- "./imgs/test_cases/two_man.jpg",
- None,
- None,
- 1024,
- 1024,
- 2.5,
- 1.6,
- 50,
- 0,
- True,
- False,
- False,
- 1024,
- ],
- [
- "Two woman are raising fried chicken legs in a bar. A woman is
<|image_1|>. The other woman is
<|image_2|>.",
- "./imgs/test_cases/mckenna.jpg",
- "./imgs/test_cases/Amanda.jpg",
- None,
- 1024,
- 1024,
- 2.5,
- 1.8,
- 50,
- 168,
- True,
- False,
- False,
- 1024,
- ],
- [
- "A man and a short-haired woman with a wrinkled face are standing in front of a bookshelf in a library. The man is the man in the middle of
<|image_1|>, and the woman is oldest woman in
<|image_2|>",
- "./imgs/test_cases/1.jpg",
- "./imgs/test_cases/2.jpg",
- None,
- 1024,
- 1024,
- 2.5,
- 1.6,
- 50,
- 60,
- True,
- False,
- False,
- 1024,
- ],
- [
- "A man and a woman are sitting at a classroom desk. The man is the man with yellow hair in
<|image_1|>. The woman is the woman on the left of
<|image_2|>",
- "./imgs/test_cases/3.jpg",
- "./imgs/test_cases/4.jpg",
- None,
- 1024,
- 1024,
- 2.5,
- 1.8,
- 50,
- 66,
- True,
- False,
- False,
- 1024,
- ],
- [
- "The flower
<|image_1|><\/img> is placed in the vase which is in the middle of
<|image_2|><\/img> on a wooden table of a living room",
- "./imgs/test_cases/rose.jpg",
- "./imgs/test_cases/vase.jpg",
- None,
- 1024,
- 1024,
- 2.5,
- 1.6,
- 50,
- 0,
- True,
- False,
- False,
- 1024,
- ],
- [
- "
<|image_1|>
\n Remove the woman's earrings. Replace the mug with a clear glass filled with sparkling iced cola.",
- "./imgs/demo_cases/t2i_woman_with_book.png",
- None,
- None,
- None,
- None,
- 2.5,
- 1.6,
- 50,
- 222,
- True,
- False,
- True,
- 1024,
- ],
- [
- "Detect the skeleton of human in this image:
<|image_1|>.",
- "./imgs/test_cases/control.jpg",
- None,
- None,
- None,
- None,
- 2.0,
- 1.6,
- 50,
- 0,
- True,
- False,
- True,
- 1024,
- ],
- [
- "Generate a new photo using the following picture and text as conditions:
<|image_1|>
\n A young boy is sitting on a sofa in the library, holding a book. His hair is neatly combed, and a faint smile plays on his lips, with a few freckles scattered across his cheeks. The library is quiet, with rows of shelves filled with books stretching out behind him.",
- "./imgs/demo_cases/skeletal.png",
- None,
- None,
- None,
- None,
- 2,
- 1.6,
- 50,
- 42,
- True,
- False,
- True,
- 1024,
- ],
- [
- "Following the pose of this image
<|image_1|>
, generate a new photo: A young boy is sitting on a sofa in the library, holding a book. His hair is neatly combed, and a faint smile plays on his lips, with a few freckles scattered across his cheeks. The library is quiet, with rows of shelves filled with books stretching out behind him.",
- "./imgs/demo_cases/edit.png",
- None,
- None,
- None,
- None,
- 2.0,
- 1.6,
- 50,
- 123,
- True,
- False,
- True,
- 1024,
- ],
- [
- "Following the depth mapping of this image
<|image_1|>
, generate a new photo: A young girl is sitting on a sofa in the library, holding a book. His hair is neatly combed, and a faint smile plays on his lips, with a few freckles scattered across his cheeks. The library is quiet, with rows of shelves filled with books stretching out behind him.",
- "./imgs/demo_cases/edit.png",
- None,
- None,
- None,
- None,
- 2.0,
- 1.6,
- 50,
- 1,
- True,
- False,
- True,
- 1024,
- ],
- [
- "
<|image_1|><\/img> What item can be used to see the current time? Please remove it.",
- "./imgs/test_cases/watch.jpg",
- None,
- None,
- None,
- None,
- 2.5,
- 1.6,
- 50,
- 0,
- True,
- False,
- True,
- 1024,
- ],
- [
- "According to the following examples, generate an output for the input.\nInput:
<|image_1|>\nOutput:
<|image_2|>\n\nInput:
<|image_3|>\nOutput: ",
- "./imgs/test_cases/icl1.jpg",
- "./imgs/test_cases/icl2.jpg",
- "./imgs/test_cases/icl3.jpg",
- 224,
- 224,
- 2.5,
- 1.6,
- 50,
- 1,
- True,
- False,
- False,
- 768,
- ],
- ]
- return case
-
-def run_for_examples(text, img1, img2, img3, height, width, guidance_scale, img_guidance_scale, inference_steps, seed, separate_cfg_infer, offload_model,
- use_input_image_size_as_output, max_input_image_size):
- return generate_image(text, img1, img2, img3, height, width, guidance_scale, img_guidance_scale, inference_steps, seed, separate_cfg_infer, offload_model,
- use_input_image_size_as_output, max_input_image_size)
-
-description = """
-OmniGen is a unified image generation model that you can use to perform various tasks, including but not limited to text-to-image generation, subject-driven generation, Identity-Preserving Generation, and image-conditioned generation.
-For multi-modal to image generation, you should pass a string as `prompt`, and a list of image paths as `input_images`. The placeholder in the prompt should be in the format of `
<|image_*|>` (for the first image, the placeholder is
<|image_1|>. for the second image, the the placeholder is
<|image_2|>).
-For example, use an image of a woman to generate a new image:
-prompt = "A woman holds a bouquet of flowers and faces the camera. Thw woman is \
\<|image_1|\>\."
-
-Tips:
-- For out of memory or time cost, you can set `offload_model=True` or refer to [./docs/inference.md#requiremented-resources](https://github.com/VectorSpaceLab/OmniGen/blob/main/docs/inference.md#requiremented-resources) to select a appropriate setting.
-- If inference time is too long when input multiple images, please try to reduce the `max_input_image_size`. More details please refer to [./docs/inference.md#requiremented-resources](https://github.com/VectorSpaceLab/OmniGen/blob/main/docs/inference.md#requiremented-resources).
-- Oversaturated: If the image appears oversaturated, please reduce the `guidance_scale`.
-- Not match the prompt: If the image does not match the prompt, please try to increase the `guidance_scale`.
-- Low-quality: More detailed prompt will lead to better results.
-- Animate Style: If the genereate images is in animate style, you can try to add `photo` to the prompt`.
-- Edit generated image. If you generate a image by omnigen and then want to edit it, you cannot use the same seed to edit this image. For example, use seed=0 to generate image, and should use seed=1 to edit this image.
-- For image editing tasks, we recommend placing the image before the editing instruction. For example, use `
<|image_1|> remove suit`, rather than `remove suit
<|image_1|>`.
-- For image editing task and controlnet task, we recommend to set the height and width of output image as the same as input image. For example, if you want to edit a 512x512 image, you should set the height and width of output image as 512x512. You also can set the `use_input_image_size_as_output` to automatically set the height and width of output image as the same as input image.
-
-
-"""
-
-article = """
----
-**Citation**
-
-If you find this repository useful, please consider giving a star ⭐ and citation
-```
-@article{xiao2024omnigen,
- title={Omnigen: Unified image generation},
- author={Xiao, Shitao and Wang, Yueze and Zhou, Junjie and Yuan, Huaying and Xing, Xingrun and Yan, Ruiran and Wang, Shuting and Huang, Tiejun and Liu, Zheng},
- journal={arXiv preprint arXiv:2409.11340},
- year={2024}
-}
-```
-**Contact**
-
-If you have any questions, please feel free to open an issue or directly reach us out via email.
-"""
-
-
-# Gradio
-with gr.Blocks() as demo:
- gr.Markdown("# OmniGen: Unified Image Generation [paper](https://arxiv.org/abs/2409.11340) [code](https://github.com/VectorSpaceLab/OmniGen)")
- gr.Markdown(description)
- with gr.Row():
- with gr.Column():
- # text prompt
- prompt_input = gr.Textbox(
- label="Enter your prompt, use <|image_i|> to represent i-th input image", placeholder="Type your prompt here..."
- )
-
- with gr.Row(equal_height=True):
- # input images
- image_input_1 = gr.Image(label="
<|image_1|>", type="filepath")
- image_input_2 = gr.Image(label="
<|image_2|>", type="filepath")
- image_input_3 = gr.Image(label="
<|image_3|>", type="filepath")
-
- # slider
- height_input = gr.Slider(
- label="Height", minimum=128, maximum=2048, value=1024, step=16
- )
- width_input = gr.Slider(
- label="Width", minimum=128, maximum=2048, value=1024, step=16
- )
-
- guidance_scale_input = gr.Slider(
- label="Guidance Scale", minimum=1.0, maximum=5.0, value=2.5, step=0.1
- )
-
- img_guidance_scale_input = gr.Slider(
- label="img_guidance_scale", minimum=1.0, maximum=2.0, value=1.6, step=0.1
- )
-
- num_inference_steps = gr.Slider(
- label="Inference Steps", minimum=1, maximum=100, value=50, step=1
- )
-
- seed_input = gr.Slider(
- label="Seed", minimum=0, maximum=2147483647, value=42, step=1
- )
-
- max_input_image_size = gr.Slider(
- label="max_input_image_size", minimum=128, maximum=2048, value=1024, step=16
- )
-
- separate_cfg_infer = gr.Checkbox(
- label="separate_cfg_infer", info="Whether to use separate inference process for different guidance. This will reduce the memory cost.", value=True,
- )
- offload_model = gr.Checkbox(
- label="offload_model", info="Offload model to CPU, which will significantly reduce the memory cost but slow down the generation speed. You can cancle separate_cfg_infer and set offload_model=True. If both separate_cfg_infer and offload_model be True, further reduce the memory, but slowest generation", value=False,
- )
- use_input_image_size_as_output = gr.Checkbox(
- label="use_input_image_size_as_output", info="Automatically adjust the output image size to be same as input image size. For editing and controlnet task, it can make sure the output image has the same size with input image leading to better performance", value=False,
- )
-
- # generate
- generate_button = gr.Button("Generate Image")
-
-
- with gr.Column():
- # output image
- output_image = gr.Image(label="Output Image")
-
- # click
- generate_button.click(
- generate_image,
- inputs=[
- prompt_input,
- image_input_1,
- image_input_2,
- image_input_3,
- height_input,
- width_input,
- guidance_scale_input,
- img_guidance_scale_input,
- num_inference_steps,
- seed_input,
- separate_cfg_infer,
- offload_model,
- use_input_image_size_as_output,
- max_input_image_size,
- ],
- outputs=output_image,
- )
-
- gr.Examples(
- examples=get_example(),
- fn=run_for_examples,
- inputs=[
- prompt_input,
- image_input_1,
- image_input_2,
- image_input_3,
- height_input,
- width_input,
- guidance_scale_input,
- img_guidance_scale_input,
- num_inference_steps,
- seed_input,
- separate_cfg_infer,
- offload_model,
- use_input_image_size_as_output,
- max_input_image_size,
- ],
- outputs=output_image,
- )
-
- gr.Markdown(article)
-
-# launch
-demo.launch()
\ No newline at end of file
diff --git a/docs/fine-tuning.md b/docs/fine-tuning.md
deleted file mode 100644
index 0dccf43..0000000
--- a/docs/fine-tuning.md
+++ /dev/null
@@ -1,172 +0,0 @@
-# Fine-tuning OmniGen
-
-Fine-tuning Omnigen can better help you handle specific image generation tasks. For example, by fine-tuning on a person's images, you can generate multiple pictures of that person while maintaining task consistency.
-
-A lot of previous work focused on designing new networks to facilitate specific tasks. For instance, ControlNet was proposed to handle image conditions, and IP-Adapter was constructed to maintain ID features. If you want to perform new tasks, you need to build new architectures and repeatedly debug them. Adding and adjusting extra network parameters is usually time-consuming and labor-intensive, which is not user-friendly and cost-efficient enough. However, with Omnigen, all of this becomes very simple.
-
-By comparison, Omnigen can accept multi-modal conditional inputs and has been pre-trained on various tasks. You can fine-tune it on any task without designing specialized networks like ControlNet or IP-Adapter for a specific task.
-
-**All you need to do is prepare the data and start training. You can break the limitations of previous models, allowing Omnigen to accomplish a variety of interesting tasks, even those that have never been done before.**
-
-
-## Installation
-
-```bash
-git clone https://github.com/VectorSpaceLab/OmniGen.git
-cd OmniGen
-pip install -e .
-```
-
-
-## Full fine-tuning
-
-### Fine-tuning command
-
-```bash
-accelerate launch \
- --num_processes=1 \
- --use_fsdp \
- --fsdp_offload_params false \
- --fsdp_sharding_strategy SHARD_GRAD_OP \
- --fsdp_auto_wrap_policy TRANSFORMER_BASED_WRAP \
- --fsdp_transformer_layer_cls_to_wrap Phi3DecoderLayer \
- --fsdp_state_dict_type FULL_STATE_DICT \
- --fsdp_forward_prefetch false \
- --fsdp_use_orig_params True \
- --fsdp_cpu_ram_efficient_loading false \
- --fsdp_sync_module_states True \
- train.py \
- --model_name_or_path Shitao/OmniGen-v1 \
- --json_file ./toy_data/toy_data.jsonl \
- --image_path ./toy_data/images \
- --batch_size_per_device 1 \
- --lr 2e-5 \
- --keep_raw_resolution \
- --max_image_size 1024 \
- --gradient_accumulation_steps 1 \
- --ckpt_every 100 \
- --epochs 100 \
- --log_every 1 \
- --results_dir ./results/toy_finetune
-```
-
-Some important arguments:
-- `num_processes`: number of GPU to use for training
-- `model_name_or_path`: path to the pretrained model
-- `json_file`: path to the json file containing the training data, e.g., ./toy_data/toy_data.jsonl
-- `image_path`: path to the image folder, e.g., ./toy_data/images
-- `batch_size_per_device`: batch size per device
-- `lr`: learning rate
-- `keep_raw_resolution`: whether to keep the original resolution of the image, if not, all images will be resized to (max_image_size, max_image_size)
-- `max_image_size`: max image size
-- `gradient_accumulation_steps`: number of steps to accumulate gradients
-- `ckpt_every`: number of steps to save checkpoint
-- `epochs`: number of epochs
-- `log_every`: number of steps to log
-- `results_dir`: path to the results folder
-
-The data format of json_file is as follows:
-```
-{
- "instruction": str,
- "input_images": [str, str, ...],
- "output_images": str
-}
-```
-You can see a toy example in `./toy_data/toy_data.jsonl`.
-
-If an OOM(Out of Memory) issue occurs, you can try to decrease the `batch_size_per_device` or `max_image_size`. You can also try to use LoRA instead of full fine-tuning.
-
-
-### Inference
-
-The checkpoint can be found at `{results_dir}/checkpoints/*`. You can use the following command to load saved checkpoint:
-```python
-from OmniGen import OmniGenPipeline
-
-pipe = OmniGenPipeline.from_pretrained("checkpoint_path") # e.g., ./results/toy_finetune/checkpoints/0000200
-```
-
-
-
-
-
-## LoRA fine-tuning
-LoRA fine-tuning is a simple way to fine-tune OmniGen with less GPU memory. To use lora, you should add `--use_lora` and `--lora_rank` to the command.
-
-```bash
-accelerate launch \
- --num_processes=1 \
- train.py \
- --model_name_or_path Shitao/OmniGen-v1 \
- --batch_size_per_device 2 \
- --condition_dropout_prob 0.01 \
- --lr 3e-4 \
- --use_lora \
- --lora_rank 8 \
- --json_file ./toy_data/toy_data.jsonl \
- --image_path ./toy_data/images \
- --max_input_length_limit 18000 \
- --keep_raw_resolution \
- --max_image_size 1024 \
- --gradient_accumulation_steps 1 \
- --ckpt_every 100 \
- --epochs 100 \
- --log_every 1 \
- --results_dir ./results/toy_finetune_lora
-```
-
-### Inference
-
-The checkpoint can be found at `{results_dir}/checkpoints/*`. You can use the following command to load checkpoint:
-```python
-from OmniGen import OmniGenPipeline
-
-pipe = OmniGenPipeline.from_pretrained("Shitao/OmniGen-v1")
-pipe.merge_lora("checkpoint_path") # e.g., ./results/toy_finetune_lora/checkpoints/0000100
-```
-
-
-## A simple example
-
-Here is an example for learning new concepts: "sks dog". We use five images of one dog from [dog-example](https://huggingface.co/datasets/diffusers/dog-example).
-
-The json file is `./toy_data/toy_subject_data.jsonl`, and the images have been saved in `./toy_data/images`.
-
-```bash
-accelerate launch \
- --num_processes=1 \
- train.py \
- --model_name_or_path Shitao/OmniGen-v1 \
- --batch_size_per_device 2 \
- --condition_dropout_prob 0.01 \
- --lr 1e-3 \
- --use_lora \
- --lora_rank 8 \
- --json_file ./toy_data/toy_subject_data.jsonl \
- --image_path ./toy_data/images \
- --max_input_length_limit 18000 \
- --keep_raw_resolution \
- --max_image_size 1024 \
- --gradient_accumulation_steps 1 \
- --ckpt_every 100 \
- --epochs 200 \
- --log_every 1 \
- --results_dir ./results/toy_finetune_lora
-```
-
-After training, you can use the following command to generate images:
-```python
-from OmniGen import OmniGenPipeline
-
-pipe = OmniGenPipeline.from_pretrained("Shitao/OmniGen-v1")
-pipe.merge_lora("checkpoint_path") # e.g., ./results/toy_finetune_lora/checkpoints/0000200
-
-images = pipe(
- prompt="a photo of sks dog running in the snow",
- height=1024,
- width=1024,
- guidance_scale=3
-)
-images[0].save("example_sks_dog_snow.png")
-```
diff --git a/docs/inference.md b/docs/inference.md
deleted file mode 100644
index 7d02e76..0000000
--- a/docs/inference.md
+++ /dev/null
@@ -1,168 +0,0 @@
-# Inference with OmniGen
-
-To handle some complex tasks, image generation models are becoming increasingly sophisticated, leading to more and more cumbersome workflows. Existing image generation models like SD and Flux require loading many additional network modules (such as ControlNet, IP-Adapter, Reference-Net) and extra preprocessing steps (e.g., face detection, pose detection, image cropping) to generate a satisfactory image. This complex workflow is not user-friendly. We believe that future image generation models should be simpler, generating various images directly through instructions, similar to how GPT works in language generation.
-
-Therefore, we propose OmniGen, a model capable of handling various image generation tasks within a single framework. The goal of OmniGen is to complete various image generation tasks without relying on any additional components or image preprocessing steps. OmniGen supports tasks including text-to-image generation, image editing, subject-driven image generation, and classical vision tasks, among others. More capabilities can be found in our examples. We provide inference code so you can explore more unknown functionalities yourself.
-
-
-
-## Install
-```bash
-git clone https://github.com/staoxiao/OmniGen.git
-cd OmniGen
-pip install -e .
-```
-
-
-
-## Generate Images
-You can use the following code to generate images:
-```python
-from OmniGen import OmniGenPipeline
-
-pipe = OmniGenPipeline.from_pretrained("Shitao/OmniGen-v1")
-
-# Text to Image
-images = pipe(
- prompt="A curly-haired man in a red shirt is drinking tea.",
- height=1024,
- width=1024,
- guidance_scale=2.5,
- seed=0,
-)
-images[0].save("example_t2i.png") # save output PIL Image
-
-# Multi-modal to Image
-# In prompt, we use the placeholder to represent the image. The image placeholder should be in the format of
<|image_*|>
-# You can add multiple images in the input_images. Please ensure that each image has its placeholder. For example, for the list input_images [img1_path, img2_path], the prompt needs to have two placeholders:
<|image_1|>,
<|image_2|>.
-images = pipe(
- prompt="A man in a black shirt is reading a book. The man is the right man in
<|image_1|>.",
- input_images=["./imgs/test_cases/two_man.jpg"],
- height=1024,
- width=1024,
- guidance_scale=2.5,
- img_guidance_scale=1.6,
- max_input_image_size=1024,
- separate_cfg_infer=True,
- use_kv_cache=True,
- offload_kv_cache=True,
- offload_model=False,
- use_input_image_size_as_output=False,
- seed=0,
-)
-images[0].save("example_ti2i.png") # save output PIL image
-```
-
-Some important arguments:
-- `guidance_scale`: The strength of the guidance. Based on our experience, it is usually best to set it between 2 and 3. The higher the value, the more similar the generated image will be to the prompt. If the image appears oversaturated, please reduce the scale.
-- `height` and `width`: The height and width of the generated image. The default value is 1024x1024. OmniGen support any size, but these number must be divisible by 16.
-- `num_inference_steps`: The number of steps to take in the diffusion process. The higher the value, the more detailed the generated image will be.
-- `max_input_image_size`: the maximum size of input image, which will be used to crop the input image to the maximum size. A smaller number will result in faster generation speed and lower memory cost.
-- `separate_cfg_infer`: Whether to use separate inference process for CFG guidance. If set to True, memory cost will be lower. Default is True.
-- `use_kv_cache`: Whether to use key-value cache. Default is True.
-- `offload_kv_cache`: offload the cached key and value to cpu, which can save memory but slow down the generation silightly. Default is True.
-- `offload_model`: offload the model to cpu, which can save memory but slow down the generation. Default is False.
-- `use_input_image_size_as_output`: whether to use the input image size as the output image size, which can be used for single-image input, e.g., image editing task. Default is False.
-- `seed`: The seed for random number generator.
-
-**More examples please refer to [inference.ipynb](../inference.ipynb)**
-
-
-#### Input data
-OmniGen can accept multi-modal input data. Specifically, you should pass two arguments: `prompt` and `input_images`.
-For text to image generation, you can pass a string as `prompt`, or pass a list of strings as `prompt` to generate multiple images.
-
-For multi-modal to image generation, you should pass a string as `prompt`, and a list of image paths as `input_images`. The placeholder in the prompt should be in the format of `
<|image_*|>`.
-For example, if you want to generate an image with a person holding a bouquet of flowers, you can pass the following prompt:
-```
-prompt = "A woman holds a bouquet of flowers and faces the camera. Thw woman is
<|image_1|>."
-input_images = ["./imgs/test_cases/liuyifei.png"]
-```
-The placeholder `<|image_1|>` will be replaced by the image at `input_images[0]`, i.e., `./imgs/test_cases/liuyifei.png`.
-
-If you want to generate multiple images, you can pass a list of prompts and a list of image paths. For example:
-```
-prompt = ["A woman holds a bouquet of flowers and faces the camera.", "A woman holds a bouquet of flowers and faces the camera. Thw woman is
<|image_1|>."]
-input_images = [[], ["./imgs/test_cases/liuyifei.png"]]
-```
-
-
-#### Gradio Demo
-We have constructed a online demo in [Huggingface](https://huggingface.co/spaces/Shitao/OmniGen).
-
-For the local gradio demo, you can run with the following command:
-```python
-python app.py
-```
-
-
-## Tips
-- For out of memory or time cost, you can refer to [./docs/inference.md#requiremented-resources](https://github.com/VectorSpaceLab/OmniGen/blob/main/docs/inference.md#requiremented-resources) to select a appropriate setting.
-- Oversaturated: If the image appears oversaturated, please reduce the `guidance_scale`.
-- Not match the prompt: If the image does not match the prompt, please try to increase the `guidance_scale`.
-- Low-quality: More detailed prompt will lead to better results.
-- Animate Style: If the genereate images is in animate style, you can try to add `photo` to the prompt`.
-- Edit generated image. If you generate a image by omnigen and then want to edit it, you cannot use the same seed to edit this image. For example, use seed=0 to generate image, and should use seed=1 to edit this image.
-- For image editing tasks, we recommend placing the image before the editing instruction. For example, use `
<|image_1|> remove suit`, rather than `remove suit
<|image_1|>`.
-- For image editing task and controlnet task, we recommend to set the height and width of output image as the same
-as input image. For example, if you want to edit a 512x512 image, you should set the height and width of output image as 512x512. You also can set the `use_input_image_size_as_output` to automatically set the height and width of output image as the same as input image.
-
-
-## Requiremented Resources
-
-We are currently experimenting with some techniques to reduce memory usage and improve speed, including `use_kv_cache, offload_kv_cache, separate_cfg_infer, offload_model`, which you can enable in the pipeline.
-The default setting is`use_kv_cache=True, offload_kv_cache=True, separate_cfg_infer=True, offload_model=False`.
-To reduce memory consumption while maintaining inference speed, quantization is also a method worth exploring and is left for future work.
-
-We conducted experiments on the A800 and RTX 3090. The memory requirements and inference times are shown in the table below. You can choose the appropriate settings based on your available resources.
-
-**Overall, the text-to-image task requires minimal memory and time costs, comparable to other latest text-to-image models. However, when using input images, the computational cost increases. If out of memory, you can set `offload_model=True`. If inference time is too long, you can reduce the `max_input_image_size`**
-
-
-- Different image size.
-
-Different image size (`max_input_image_size` is the max size of input image, `height` and `width` are the size of output image) with the default inference settings (`use_kv_cache=True,offload_kv_cache=True,separate_cfg_infer=True`)
-
-For A800 GPU:
-| Settings | Only Text | Text + Single Image | Text + Two Images |
-|:-------------|:----------:|:-------------------:|:---------------------:|
-| max_input_image_size=1024,height=1024,width=1024 | 9G, 31s | 12G, 1m6s | 13G, 1m20s |
-| max_input_image_size=512,height=1024,width=1024 | 9G, 31s | 10G, 50s | 10G, 54s |
-| max_input_image_size=768,height=768,width=768 | 9G, 16s | 10G, 32s | 10G, 37s |
-| max_input_image_size=512,height=512,width=512 | 9G, 7s | 9G, 14s | 9G, 15s |
-
-For RTX 3090 GPU(24G):
-| Settings | Only Text | Text + Single Image | Text + Two Images |
-|:-------------|:----------:|:-------------------:|:---------------------:|
-| max_input_image_size=1024,height=1024,width=1024 | 9G, 1m17s | 12G, 2m46s | 13G, 3m23s |
-| max_input_image_size=512,height=1024,width=1024 | 9G, 1m18s | 10G, 2m8s | 10G, 2m18s |
-| max_input_image_size=768,height=768,width=768 | 9G, 41s | 10G, 1m22s | 10G, 1m38s |
-| max_input_image_size=512,height=512,width=512 | 9G, 19s | 9G, 36s | 9G, 43s |
-
-
-We recommend reducing the size of input images to improve speed (i.e., reduce the `max_input_image_size`), especially when inputting multiple images, as multiple large images can significantly slow down the process.
-A very smaller `max_input_image_size` to significantly reduce memory usage and speed-up generation, but note that the generation quality may be lower.
-And please set the `height` and `width` the same as the size of input image for image editing task.
-
-
-- Different inference settings
-
-Default image size: height=1024, width=1024, max_input_image_size=1024
-
-For A800 GPU:
-| Settings | Only Text | Text + Single Image | Text + Two Images |
-|:-------------|:----------:|:-------------------:|:---------------------:|
-| use_kv_cache | 18G, 30s | 36G, 1m | 48G, 1m13s |
-| use_kv_cache,offload_kv_cache | 10G, 30s | 14G, 1m10s | 17G, 1m30s |
-| use_kv_cache,offload_kv_cache,separate_cfg_infer | 9G, 31s | 12G, 1m6s | 13G, 1m20s |
-| use_kv_cache,offload_kv_cache,offload_model | 4G, 55s | 7G, 1m30s | 11G, 1m48s |
-| use_kv_cache,offload_kv_cache,separate_cfg_infer,offload_model | 3G, 1m23s | 5G, 2m19s | 6G, 2m30s |
-
-For RTX 3090 GPU(24G):
-| Settings | Only Text | Text + Single Image | Text + Two Images |
-|:-------------|:----------:|:-------------------:|:---------------------:|
-| use_kv_cache | 18G, 1m14s | OOM | OOM |
-| use_kv_cache,offload_kv_cache | 10G, 1m17s | 14G, 3m11s | 17G, 4m3s |
-| use_kv_cache,offload_kv_cache,separate_cfg_infer | 9G, 1m18s | 12G, 2m46s | 13G, 3m21s |
-| use_kv_cache,offload_kv_cache,offload_model | 4G,3m1s | 7G, 4m14s | 11G, 5m4s |
-| use_kv_cache,offload_kv_cache,separate_cfg_infer,offload_model | 3G, 4m56s | 5G, 7m49s | 6G, 8m6s |
diff --git a/imgs/.DS_Store b/img/.DS_Store
similarity index 92%
rename from imgs/.DS_Store
rename to img/.DS_Store
index 5008ddf..4dff23b 100644
Binary files a/imgs/.DS_Store and b/img/.DS_Store differ
diff --git a/img/OmniGen-demo-video.mov b/img/OmniGen-demo-video.mov
new file mode 100644
index 0000000..8df5629
Binary files /dev/null and b/img/OmniGen-demo-video.mov differ
diff --git a/img/X2I.png b/img/X2I.png
new file mode 100644
index 0000000..2b72fbd
Binary files /dev/null and b/img/X2I.png differ
diff --git a/toy_data/images/cat.png b/img/common_tasks.png
similarity index 52%
rename from toy_data/images/cat.png
rename to img/common_tasks.png
index 7de734c..b741f8d 100644
Binary files a/toy_data/images/cat.png and b/img/common_tasks.png differ
diff --git a/img/cv.png b/img/cv.png
new file mode 100644
index 0000000..1f81969
Binary files /dev/null and b/img/cv.png differ
diff --git a/imgs/demo_cases.png b/img/demo_cases.png
similarity index 100%
rename from imgs/demo_cases.png
rename to img/demo_cases.png
diff --git a/img/framework.png b/img/framework.png
new file mode 100644
index 0000000..8435ad6
Binary files /dev/null and b/img/framework.png differ
diff --git a/imgs/overall.jpg b/img/overall.jpg
similarity index 100%
rename from imgs/overall.jpg
rename to img/overall.jpg
diff --git a/img/reason_icl.png b/img/reason_icl.png
new file mode 100644
index 0000000..bebe8fc
Binary files /dev/null and b/img/reason_icl.png differ
diff --git a/img/referring.png b/img/referring.png
new file mode 100644
index 0000000..efb5e26
Binary files /dev/null and b/img/referring.png differ
diff --git a/img/step.png b/img/step.png
new file mode 100644
index 0000000..d145ee6
Binary files /dev/null and b/img/step.png differ
diff --git a/imgs/demo_cases/AI_Pioneers.jpg b/imgs/demo_cases/AI_Pioneers.jpg
deleted file mode 100644
index e77c077..0000000
Binary files a/imgs/demo_cases/AI_Pioneers.jpg and /dev/null differ
diff --git a/imgs/demo_cases/edit.png b/imgs/demo_cases/edit.png
deleted file mode 100644
index 72c8967..0000000
Binary files a/imgs/demo_cases/edit.png and /dev/null differ
diff --git a/imgs/demo_cases/entity.png b/imgs/demo_cases/entity.png
deleted file mode 100644
index 355d005..0000000
Binary files a/imgs/demo_cases/entity.png and /dev/null differ
diff --git a/imgs/demo_cases/reasoning.png b/imgs/demo_cases/reasoning.png
deleted file mode 100644
index cee6b91..0000000
Binary files a/imgs/demo_cases/reasoning.png and /dev/null differ
diff --git a/imgs/demo_cases/same_pose.png b/imgs/demo_cases/same_pose.png
deleted file mode 100644
index 6c10c6a..0000000
Binary files a/imgs/demo_cases/same_pose.png and /dev/null differ
diff --git a/imgs/demo_cases/skeletal.png b/imgs/demo_cases/skeletal.png
deleted file mode 100644
index 145fe15..0000000
Binary files a/imgs/demo_cases/skeletal.png and /dev/null differ
diff --git a/imgs/demo_cases/skeletal2img.png b/imgs/demo_cases/skeletal2img.png
deleted file mode 100644
index d7f8777..0000000
Binary files a/imgs/demo_cases/skeletal2img.png and /dev/null differ
diff --git a/imgs/demo_cases/t2i_woman_with_book.png b/imgs/demo_cases/t2i_woman_with_book.png
deleted file mode 100644
index 44bfbef..0000000
Binary files a/imgs/demo_cases/t2i_woman_with_book.png and /dev/null differ
diff --git a/imgs/referring.png b/imgs/referring.png
deleted file mode 100644
index bba066b..0000000
Binary files a/imgs/referring.png and /dev/null differ
diff --git a/imgs/test_cases/1.jpg b/imgs/test_cases/1.jpg
deleted file mode 100644
index 5233e29..0000000
Binary files a/imgs/test_cases/1.jpg and /dev/null differ
diff --git a/imgs/test_cases/2.jpg b/imgs/test_cases/2.jpg
deleted file mode 100644
index 1344359..0000000
Binary files a/imgs/test_cases/2.jpg and /dev/null differ
diff --git a/imgs/test_cases/3.jpg b/imgs/test_cases/3.jpg
deleted file mode 100644
index a5e4d10..0000000
Binary files a/imgs/test_cases/3.jpg and /dev/null differ
diff --git a/imgs/test_cases/4.jpg b/imgs/test_cases/4.jpg
deleted file mode 100644
index 563dddf..0000000
Binary files a/imgs/test_cases/4.jpg and /dev/null differ
diff --git a/imgs/test_cases/Amanda.jpg b/imgs/test_cases/Amanda.jpg
deleted file mode 100644
index 63c3cb1..0000000
Binary files a/imgs/test_cases/Amanda.jpg and /dev/null differ
diff --git a/imgs/test_cases/control.jpg b/imgs/test_cases/control.jpg
deleted file mode 100644
index 71c0d75..0000000
Binary files a/imgs/test_cases/control.jpg and /dev/null differ
diff --git a/imgs/test_cases/icl1.jpg b/imgs/test_cases/icl1.jpg
deleted file mode 100644
index fc7e56d..0000000
Binary files a/imgs/test_cases/icl1.jpg and /dev/null differ
diff --git a/imgs/test_cases/icl2.jpg b/imgs/test_cases/icl2.jpg
deleted file mode 100644
index 818bc93..0000000
Binary files a/imgs/test_cases/icl2.jpg and /dev/null differ
diff --git a/imgs/test_cases/icl3.jpg b/imgs/test_cases/icl3.jpg
deleted file mode 100644
index a1d2cea..0000000
Binary files a/imgs/test_cases/icl3.jpg and /dev/null differ
diff --git a/imgs/test_cases/lecun.png b/imgs/test_cases/lecun.png
deleted file mode 100644
index ca377e1..0000000
Binary files a/imgs/test_cases/lecun.png and /dev/null differ
diff --git a/imgs/test_cases/mckenna.jpg b/imgs/test_cases/mckenna.jpg
deleted file mode 100644
index 139eedd..0000000
Binary files a/imgs/test_cases/mckenna.jpg and /dev/null differ
diff --git a/imgs/test_cases/pose.png b/imgs/test_cases/pose.png
deleted file mode 100644
index 1805e1f..0000000
Binary files a/imgs/test_cases/pose.png and /dev/null differ
diff --git a/imgs/test_cases/rose.jpg b/imgs/test_cases/rose.jpg
deleted file mode 100644
index 5cbd8fa..0000000
Binary files a/imgs/test_cases/rose.jpg and /dev/null differ
diff --git a/imgs/test_cases/trump.png b/imgs/test_cases/trump.png
deleted file mode 100644
index d230f3f..0000000
Binary files a/imgs/test_cases/trump.png and /dev/null differ
diff --git a/imgs/test_cases/turing.png b/imgs/test_cases/turing.png
deleted file mode 100644
index e77c077..0000000
Binary files a/imgs/test_cases/turing.png and /dev/null differ
diff --git a/imgs/test_cases/two_man.jpg b/imgs/test_cases/two_man.jpg
deleted file mode 100644
index 384fe95..0000000
Binary files a/imgs/test_cases/two_man.jpg and /dev/null differ
diff --git a/imgs/test_cases/vase.jpg b/imgs/test_cases/vase.jpg
deleted file mode 100644
index 92f4c64..0000000
Binary files a/imgs/test_cases/vase.jpg and /dev/null differ
diff --git a/imgs/test_cases/watch.jpg b/imgs/test_cases/watch.jpg
deleted file mode 100644
index d751287..0000000
Binary files a/imgs/test_cases/watch.jpg and /dev/null differ
diff --git a/imgs/test_cases/woman.png b/imgs/test_cases/woman.png
deleted file mode 100644
index 0d3e8b3..0000000
Binary files a/imgs/test_cases/woman.png and /dev/null differ
diff --git a/imgs/test_cases/yifei2.png b/imgs/test_cases/yifei2.png
deleted file mode 100644
index 8ea709f..0000000
Binary files a/imgs/test_cases/yifei2.png and /dev/null differ
diff --git a/imgs/test_cases/young_musk.jpg b/imgs/test_cases/young_musk.jpg
deleted file mode 100644
index 1dc3582..0000000
Binary files a/imgs/test_cases/young_musk.jpg and /dev/null differ
diff --git a/imgs/test_cases/young_trump.jpeg b/imgs/test_cases/young_trump.jpeg
deleted file mode 100644
index 7b1d0f3..0000000
Binary files a/imgs/test_cases/young_trump.jpeg and /dev/null differ
diff --git a/imgs/test_cases/zhang.png b/imgs/test_cases/zhang.png
deleted file mode 100644
index 4bda395..0000000
Binary files a/imgs/test_cases/zhang.png and /dev/null differ
diff --git a/index.html b/index.html
new file mode 100644
index 0000000..12d1e00
--- /dev/null
+++ b/index.html
@@ -0,0 +1,487 @@
+
+
+
+
+ The emergence of Large Language Models (LLMs) has unified language generation tasks and + revolutionized human-machine interaction. However, in the realm of image generation, a + unified model capable of handling various tasks within a single framework remains largely + unexplored. In this work, we introduce OmniGen, a new diffusion model for unified image + generation. Unlike popular diffusion models (e.g., Stable Diffusion), OmniGen no longer + requires additional modules such as ControlNet or IP-Adapter to process diverse control + conditions. + + OmniGen is characterized by the following features: + +
Based on OmniGen's general capabilities, more flexible image generation can be implemented. + The following showcases a simple pipeline: generating images from text, editing parts of the generated images, + generating redraws based on the human poses in the generated images, + and extracting desired objects from another image to integrate into the new image. +
+You can input multiple images and use simple, general language to refer to the objects within those images. + OmniGen can automatically recognize the necessary objects in each image and generate new images based on them. + No additional operations, such as image cropping or face detection, are required. +
+OmniGen can process various image generation tasks, inlcuding image editing, image-conditional gengeration (controlnet), etc. +
+OmniGen also is able to process some classical computer vision tasks, e.g., low-level tasks: deblur, derain, inpainting; high level tasks: human pose estimation, depth estimation. +
+OmniGen has potential inference capabilities and a certain degree of in-context learning (ICL) ability. +
+The Chain-of-Thought (CoT) method can significantly boost the performance of LLMs by decomposing the + task into multiple steps and sequentially solving each step to obtain an accurate final answer. + We consider whether a similar alternative can be applied to image generation. + Inspired by the basic way of human drawing, we hope to mimic the step-by-step drawing process, + iteratively refining theimage from a blank canvas. We fine-tune the OmniGen to process this task. + Based on the findings of previous work on LLMs, which indicate that process supervision significantly + outperforms outcome supervision, we posit that supervising the drawing process of images is a + promising direction that may assist the model in handling more complex and diverse scenes. +
++ Current diffusion models are typically limited to common text-to-image tasks and can not + perform a broader range of downstream image-generation tasks. To achieve real-world applications, + users often need to design and integrate additional network structures to extend the capabilities of + diffusion models, making the models highly cumbersome. Even worse, these additional parameter + networks are usually task-specific and can not be reused for other tasks, unless more networks + are designed and trained for different functions. To circumvent these issues, the design principles + of OmniGen are as follows: 1). Universality: accepting any form of image and text inputs for + various tasks; 2). Conciseness, avoiding overly complex structural designs and numerous additional + components. + +
+To achieve robust multi-task processing capabilities, it is essential to train models on large-scale and + diverse datasets. However, in the field of image generation, a readily available large-scale and diverse + dataset has yet to emerge. In this work, we have constructed a large-scale unified image generation + dataset for the first time, which we refer to as the X2I dataset, meaning "anything to image". We + have converted these data into a unified format, and following figure presents some examples from the X2I + dataset. The entire dataset comprises approximately 0.1 billion images. We will provide a detailed + description of the composition of this dataset in the following sections. +
+
+@misc{omnigen2024,
+ author={Shitao Xiao, Yueze Wang, Junjie Zhou, Huaying Yuan, Xingrun Xing, Ruiran Yan, Shuting Wang, Tiejun Huang, Zheng Liu},
+ title={OmniGen: Unified Image Generation},
+ publisher={arXiv:2409.11340v1},
+ year={2024},
+}
+
+