VATE

---

Quick Links

• Overview
• Repository Structure
• Getting Started
  • Installation
  • Downloading VATE
  • Contrastive model
• Contributing
• License

---

Overview

the Video-Audio-Text for affective Evaluation dataset. VATE collects a wide variety of multimodal data exhibiting a multitude of spontaneous human affective states. It contains 21,871 raw videos together with voice recordings and text transcriptions from numerous emotion evoking interviews. VATE is specifically designed for contrastive self-supervised representation learning of human affective states; it prioritises quantity and quality of data over human labelling of emotions, which constitutes a highly subjective, often inconsistent and controversial aspect of modern affective computing. To highlight the usefulness of our proposal, we release a multimodal encoder employing a contrastive video-language-audio pre-training procedure carried out on the VATE dataset. Experimental results show that such model exhibits sensibly better few-shot generalisation abilities when compared to fully supervised baselines on different downstream tasks.

An overview of the data can be found at

VATE/output/VATE/metadata.json

---

Repository Structure

└── VATE/
    ├── VATE.py
    ├── README.md
    ├── audio.py
    ├── contrastive_model.py
    ├── dataset.py
    ├── dataset_utils.py
    ├── feature_extraction
    │   ├── VATE
    │   ├── collect_yb.py
    │   ├── couples.txt
    │   ├── cut_video.py
    │   ├── input.txt
    │   ├── main.py
    │   └── write_video.py
    ├── main.py
    ├── media.py
    ├── output
    │   └── VATE
    │       ├── best_model_contrastive.pt
    │       └── metadata.json
    ├── text.py
    ├── train_test.py
    ├── utils.py
    └── video.py

---

Getting Started

Installation

1. Clone the VATE repository:

git clone https://github.com/FrancescoAgnelli3/VATE

2. Change to the project directory:

cd VATE

3. Install the dependencies:

pip install -r requirements.txt

Downloading VATE

Use the following command to download the VATE dataset:

1. Change to the project directory:

cd feature_extraction

2. Download the dataset:

python main.py

The dataset will be downloaded in the folder:

VATE/feature_extraction/VATE

If you want to add other YouTube playlists to the dataset, you can add them to the python file and run:

python collect_yb.py

And then again:

python main.py

Contrastive model

1. To train the contrastive model on the dataset, change to the project directory:

cd ..

2. Train the model:

python main.py

3. The model will be saved in (or it can be directly downloaded, already pre-trained, from) the folder:

VATE/output/VATE/best_model_contrastive.pt

Contributing

To contribute to the project, please follow these guidelines:

1. Fork the repository and clone it to your local machine.

2. Create a new branch for your feature or bug fix.

3. Make your changes and commit them with descriptive commit messages.

4. Push your branch to your forked repository.

5. Submit a pull request to the main repository.

License

This project is protected under the MIT LICENSE License.