DreamEdit: Subject-driven Image Editing

Replace the subject in a given image to a customized one or add your customized subject to any provided background!

Models, code, and dataset for DreamEdit: Subject-driven Image Editing.

Check project website for demos and data examples.

News

• [26/10/2023] Paper accepted to TMLR 2023.
• [20/07/2023] We released our DreamBooth weights on UWaterloo Vault and HuggingFace Space.

Requirements

A suitable conda environment named dream_edit can be created and activated with:

conda env create -f environment.yml
conda activate dream_edit

# To update env
conda env update dream_edit --file environment.yml  --prune

^There is some problem with the environment file setup currently, we will fix it soon.

For now to get the code able to run: Our repo requires dependencies from different repos. Please follow the official installation of:

• GroundingDINO
• LangSAM
• Stable Diffusion
• Gligen's fork of diffuser

For example, besides the auto install environment, we also install dependencies with:

pip install torch==1.12.1+cu116 torchvision==0.13.1+cu116 torchaudio==0.12.1 --extra-index-url https://download.pytorch.org/whl/cu116
cd ..

# Install SAM
pip install git+https://github.com/luca-medeiros/lang-segment-anything.git (already included in the yml file)
git clone https://github.com/IDEA-Research/Grounded-Segment-Anything.git

# To enable gpu in grounding dino:
conda install -c conda-forge cudatoolkit-dev -y
export BUILD_WITH_CUDA=True
export CUDA_HOME=$CONDA_PREFIX
export AM_I_DOCKER=False

# This might be optional:
cd ~/dreamedit_env_dependency/Grounded-Segment-Anything/
python -m pip install -e segment_anything
python -m pip install -e GroundingDINO
git submodule update --init --recursive
cd grounded-sam-osx && bash install.sh

pip install opencv-python pycocotools matplotlib onnxruntime onnx ipykernel
pip install accelerate

# Install diffusers in gligen fork to enable gligen pipeline:
git clone https://github.com/gligen/diffusers.git
cd diffusers
pip install -e .

Huggingface Dataset

Our dataset is on Huggingface now: https://huggingface.co/datasets/tianleliphoebe/DreamEditBench, a more self-contained one: https://huggingface.co/datasets/tianleliphoebe/DreamEditBench_SelfContained.

from datasets import load_dataset
dataset = load_dataset("tianleliphoebe/DreamEditBench")

How to run

Go to experiment-results-analysis folder:

cd experiment-results-analysis/experiments

Run the script:

sh replace_dog8_config_01.sh

You can change the input path for data, model, and other parameter setting in the corresponding config file (e.g. replace_dog8_config_01.yaml). The data: src_img_data_folder_path should be specified as the path to our DreamEditBench dataset can be downloaded as mentioned above. And the db_dataset_path should be the path of the original dreambooth dataset that can be downloaded here: https://github.com/google/dreambooth/tree/main/dataset. An example fine-tuned dreambooth model checkpoint for dog8 can be downloaded at here.

All the other subject fine-tuned model weights can be downloaded at this link. You can also finetune your own dreambooth models following the implementation.

BibTeX

If you find this paper or repo useful for your research, please consider citing our paper:

@misc{li2023dreamedit,
      title={DreamEdit: Subject-driven Image Editing}, 
      author={Tianle Li and Max Ku and Cong Wei and Wenhu Chen},
      year={2023},
      eprint={2306.12624},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Star History