Skip to content

Latest commit

 

History

History
64 lines (51 loc) · 2.08 KB

README.md

File metadata and controls

64 lines (51 loc) · 2.08 KB

STDiff applied to CLEVRER

adapted from XiYe20/STDiffProject (paper: arXiv link)

STDiff Architecture

Overview

STDiff Architecture

Installation

  1. Install the custom diffusers library
git clone https://github.com/XiYe20/CustomDiffusers.git
cd CustomDiffusers
pip install -e .
  1. Install the requirements
pip install -r requirements.txt

Dataset

The dataset consists of 13000 video clips of 22 frames each in the unlabeled folder. The val and train folder are for inference.

available here: https://drive.google.com/file/d/1iYTFuf4DgxgYQzTQ_2da1vC9es_niPRr/view?usp=drive_link

Folder Structure
     unlabeled/
         video_02000/
            image_0.png
            image_1.png
            ...
            image_21.png
         video_02001/
            ...
         video_...
     train/
         ...
     val/
         ...

Note that there are also masks.npy files which are meant for segmentation.

Training and Evaluation

Simiilar to STDiff project, accelerate is used for training. The configuration files are placed inside stdiff/configs.

Training

  1. Check train.sh, modify the visible gpus, num_process, modify the config.yaml file
  2. Training
. ./train.sh

Test

  1. Check inference.sh, modify config.yaml for inference
  2. Test
. ./inference.sh