STDiff applied to CLEVRER

adapted from XiYe20/STDiffProject (paper: arXiv link)

Overview

Installation

1. Install the custom diffusers library

git clone https://github.com/XiYe20/CustomDiffusers.git
cd CustomDiffusers
pip install -e .

2. Install the requirements

pip install -r requirements.txt

Dataset

The dataset consists of 13000 video clips of 22 frames each in the unlabeled folder. The val and train folder are for inference.

available here: https://drive.google.com/file/d/1iYTFuf4DgxgYQzTQ_2da1vC9es_niPRr/view?usp=drive_link

Folder Structure
     unlabeled/
         video_02000/
            image_0.png
            image_1.png
            ...
            image_21.png
         video_02001/
            ...
         video_...
     train/
         ...
     val/
         ...

Note that there are also masks.npy files which are meant for segmentation.

Training and Evaluation

Simiilar to STDiff project, accelerate is used for training. The configuration files are placed inside stdiff/configs.

Training

1. Check train.sh, modify the visible gpus, num_process, modify the config.yaml file
2. Training

. ./train.sh

Test

1. Check inference.sh, modify config.yaml for inference
2. Test

. ./inference.sh