X-Prompt: Multi-modal Visual Prompt for Video Object Segmentation

This repository contains the code for the paper X-Prompt: Multi-modal Visual Prompt for Video Object Segmentation [ACMMM'2024].

Install

conda create -n vos python=3.9 -y
conda activate vos
pip install torch==1.13.1+cu117 torchvision==0.14.1+cu117 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/cu117

pip install -r requirements.txt

git clone https://github.com/ClementPinard/Pytorch-Correlation-extension.git
cd Pytorch-Correlation-extension
python setup.py install
cd -

Prepare

Dataset

Reformed and preprocessed datasets that can be used directly:

• [RGB-T] VisT300
• [RGB-T] VT-UVA
• [RGB-D] ARKitTrack
• [RGB-E] Since the VisEvent dataset only provides bounding box annotations and lacks mask annotations, we initially used HQ-SAM to generate masks for our experiments. Now we notice a new RGB-E dataset LLE-VOS annoted with masks for VOS has been released. We recommend using LLE-VOS directly, and we will be updating our work with experiments on the RGB-E data from LLE-VOS.

├── X-Prompt
├── datasets
│   ├── VisT300
│   │    └── ...
│   ├── VT-UVA
│   │    └── ...
│   ├── ARKitTrack
│   │    └── ...
│   ├── ...

Pretrained Foudantion Model

• Pretrained RGB VOS Foundation Model

Place the weights in X-Prompt/weights. We pretrain the RGB foundation model following the OneVOS [ECCV'2024]. More details on the model design and training, please refer to OneVOS.

Train & Eval

# [RGB-T] VisT300 and VT-UAV
bash exp_rgbt.sh

# [RGB-D] ARKitTrack
bash exp_arkittrack.sh

We also provide our trained models you can use directly for inference and evaluation.

• X-Prompt RGB-T
• X-Prompt RGB-D

Note that we didn't thoroughly search for hyperparameters, so there may be better choices to get better performance than reported.

Citations

If you find this repository useful, please consider giving a star and citation:

@inproceedings{guo2024x,
  title={X-Prompt: Multi-modal Visual Prompt for Video Object Segmentation},
  author={Guo, Pinxue and Li, Wanyun and Huang, Hao and Hong, Lingyi and Zhou, Xinyu and Chen, Zhaoyu and Li, Jinglun and Jiang, Kaixun and Zhang, Wei and Zhang, Wenqiang},
  booktitle={ACM Multimedia 2024}
}

@article{li2024onevos,
  title={OneVOS: Unifying Video Object Segmentation with All-in-One Transformer Framework},
  author={Li, Wanyun and Guo, Pinxue and Zhou, Xinyu and Hong, Lingyi and He, Yangji and Zheng, Xiangyu and Zhang, Wei and Zhang, Wenqiang},
  journal={arXiv preprint arXiv:2403.08682},
  year={2024}
}