RORem: Training a Robust Object Remover with Human-in-the-Loop

Ruibin Li^1,2 | Tao Yang³ | Song Guo⁴ | Lei Zhang^1,2 |

¹The Hong Kong Polytechnic University, ²OPPO Research Institute, ³ByteDance, ⁴The Hong Kong University of Science and Technology.

⏰ Update

The code and model will be ready soon.

⭐: If RORem is helpful to you, please help star this repo. Thanks! 🤗

🌟 Overview Framework

Overview of our training data generation and model training process. In stage 1, we gather 60K training triplets from open-source datasets to train an initial removal model. In stage 2, we apply the trained model to a test set and engage human annotators to select high-quality samples to augment the training set. In stage 3, we train a discriminator using the human feedback data, and employ it to automatically annotate high quality training samples. We iterate stages 2&3 for several rounds, ultimately obtaining over 200K object removal training triplets as well as the trained model.

🌟 Visual Results

Quantative comparsion

We invite human annotators to evaluate the success rate of different methods. Furthermore, by refining our discriminator, we can see that the success rates estimated by $D_{\phi}$ closely align with human annotations in the test set (with deviations less than 3% in most cases). This indicates that our trained $D_{\phi}$ effectively mirrors human preferences.

Qualitative Comparisons

License

This project is released under the Apache 2.0 license.

BibTeX

@article{li2024RORem,
  title={RORem: Training a Robust Object Remover with Human-in-the-Loop},
  author={Ruibin Li and Tao, Yang and Song, Guo and Lei, Zhang},
  year={2024},
  journal={arXiv preprint arXiv:2501.00740},
}

statistics