Dataset and Structure

Datasets

MMSA-FET currently supports three MSA datasets: CMU-MOSI, CMU-MOSEI and CH-SIMS. The raw videos, feature files and label files can be downloaded from the following links.

• BaiduYun Disk
• Google Drive

You can add your own dataset as long as your dataset folder follow the structure described below.

Folder Structure & Label File

The datasets need to be organized in a specific folder structure for MMSA-FET to work properly on them.

• The dataset files should be organized as follows:

  .
  ├── DatasetName
  │   ├── label.csv           // label file
  │   ├── Processed
  │   │   ├── feature_1.pkl   // feature files
  │   │   ├── feature_2.pkl
  │   │   └── features.pkl
  │   ├── Raw
  │   │   ├── video_001       // video_id
  │   │   │   ├── 0001.mp4    // clip_id
  │   │   │   ├── 0002.mp4
  │   │   │   ├── 0003.mp4
  │   │   │   └── 0004.mp4
  │   │   ├── video_002       // video_id
  │   │   │   ├── 0001.mp4    // clip_id
  │   │   │   ├── 0002.mp4
  │   │   ├── video_003
  ...
  

• The label.csv file should contain columns in exact same order as the below example. The example corresponds to the above folder structure.

  video_id,clip_id,text,label,label_T,label_A,label_V,annotation,mode
  video_id  ,clip_id ,text                               ,label ,label_T ,label_A ,label_V ,annotation ,mode
  video_001 ,0001    ,A LOT OF SAD PARTS                 ,-0.5  ,        ,        ,        ,Neutral    ,train
  video_001 ,0002    ,THERE IS SAD PART                  ,-1.2  ,        ,        ,        ,Negative   ,train
  video_001 ,0003    ,AND ITS A REALLY FUNNY             ,1.8   ,        ,        ,        ,Positive   ,train
  video_001 ,0004    ,BUT IT WAS REALLY REALLY AWESOME   ,2.2   ,        ,        ,        ,Positive   ,train
  video_002 ,0001    ,ANYHOW IT WAS REALLY GOOD          ,2.4   ,        ,        ,        ,Positive   ,train
  video_002 ,0002    ,I MEAN THEY DID A LITTLE BIT OF IT ,-1.0  ,        ,        ,        ,Negative   ,train
  

Feature File

MMSA-FET generates feature pickle files which are compatible with MMSA. The feature file is organized as described in MMSA wiki