ReNeg: Learning Negative Embedding with Reward Guidance

 

We present ReNeg, a Reward-guided approach that directly learns Negative embeddings through gradient descent. The global negative embeddings learned using ReNeg exhibit strong generalization capabilities and can be seamlessly adaptable to text-to-image and even text-to-video models. Strikingly simple yet highly effective, ReNeg amplifies the visual appeal of outputs from base Stable Diffusion models.

If you find ReNeg's open-source effort useful, please 🌟 us to encourage our following development!

🔧 Installation

conda create -n reneg python=3.8.5
pip install torch==2.0.0 torchvision==0.15.1 torchaudio==2.0.1 --index-url https://download.pytorch.org/whl/cu118

# Clone the ReNeg repository 
git clone https://github.com/XiaominLi1997/ReNeg.git
cd ReNeg
pip install -r requirements.txt

🗄️ Models and Demos

Any text-conditioned generative model utilizing the same text encoder can share their negative embeddings. We provide the following ReNeg embeddings of common text encoders.

1. Models

Text Encoder	Model	Path
CLIP ViT-L/14	SD1.4	sd1.4_reneg_emb
	SD1.5	sd1.5_reneg_emb
OpenCLIP-ViT/H	SD2.1	sd2.1_reneg_emb
T5-v1.1-xxl	Pixart-alpha	pixart-alpha_reneg_emb

2. Demos

Negative Embeddings of SD and Pixart-alpha:

Text Encoder	Model	Results
CLIP ViT-L/14	SD1.4
	SD1.5
OpenCLIP-ViT/H	SD2.1
T5-v1.1-xxl	Pixart-alpha

Transfer of Negative Embeddings:

Text Encoder	Transfer of Neg. Emb.	Results
OpenCLIP-ViT/H	SD2.1 -> ZeroScope
	SD2.1 -> VideoCrafter2
T5-v1.1-xxl	Pixart-alpha -> LTX-Video

💻 Inference

You need to first specify the paths for SD1.5 and neg_emb. By default, we place neg_emb under the checkpoints/ directory.

python inference.py --model_path "your_sd1.5_path" --neg_embeddings_path "checkpoints/checkpoint.bin" --prompt "A girl in a school uniform playing an electric guitar."

To compare with the inference results using neg_emb, you can perform inference using only positive prompt, or use a specific negative prompt.

• To perform inference using only the pos_prompt, you need to specify args.prompt_type = only_pos.

python inference.py --model_path "your_sd1.5_path" --prompt_type "only_pos" --prompt "A girl in a school uniform playing an electric guitar."

• To perform inference using pos_prompt + neg_prompt, example negative prompts include: distorted, ugly, blurry, low resolution, low quality, bad, deformed, disgusting, Overexposed, Simple background, Plain background, Grainy, Underexposed, too dark, too bright, too low contrast, too high contrast, Broken, Macabre, artifacts, oversaturated

python inference.py --model_path "your_sd1.5_path" --prompt_type "neg_prompt" --prompt "A girl in a school uniform playing an electric guitar."

📋 Todo List

• Inference code
• Training code
• Online Demo

❤️ Acknowledgements

This project is based on ImageReward and diffusers. Thanks for their awesome works.

Citation

@misc{li2024reneg,
      title={ReNeg: Learning Negative Embedding with Reward Guidance},
      author={Xiaomin Li, Yixuan Liu, Takashi Isobe, Xu Jia, Qinpeng Cui, Dong Zhou, Dong Li, You He, Huchuan Lu, Zhongdao Wang, Emad Barsoum},
      year={2024},
      eprint={2412.19637},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}