Hybrid Fusion Based Approach for Multimodal Emotion Recognition with Insufficient Labeled Data

Implementation for the paper (ICIP 2021). The paper has been accepted, its full-text will be shared after publication.
Hybrid Fusion Based Approach for Multimodal Emotion Recognition with Insufficient Labeled Data
Puneet Kumar, Vedanti Khokher, Yukti Gupta, and Balasubramanian Raman

Setup and Dependencies

1. Install Anaconda or Miniconda distribution and create a conda environment with Python 3.8+.
2. Install the requirements using the following command:

pip install -r Requirements.txt

3. Download the BT4SA dataset and keep in data_t4sa folder.
4. Download glove.twitter.27B.200d.txt and keep in data_files folder.
5. Rest of the data files are already provided in the data_files folder.

Steps to run the Code

1. Text Emotion Recognition Phase:
   Run TER.ipynb in Jupyter Notebook
   OR
   Run TER.py in the terminal/command-line using the following command:

python TER.py --epoch=100

Reference: The code from TER phase has been referred from here.

2. Image Emotion Recognition Phase:
   Run IER.ipynb in Jupyter Notebook
   OR
   Run IER.py in the terminal/command-line using the following command:

python IER.py --epoch=100

Reference: This code served as an inspiration for building the code for the IER phase.

3. Intermediate Fusion Phase:
   Run Intermediate_fusion.ipynb in Jupyter Notebook
   OR
   Run Intermediate_fusion.py in the terminal/command-line using the following command:

python Intermediate_fusion.py --epoch=100

4. Late Fusion Phase:
   Run Late_fusion.ipynb OR python Late_fusion.py.

Saving Model Checkpoints

By default, the code saves the model checkpoints in the model_checkpoints folder. Troubleshooting: Sometimes the 'Kernel dead' error is caused if the model checkpoints are not properly saved or loaded.

Tensorboard Logging

The tensorboard log files are saved in the tb_logs folder for IER, TER and Intermediate_fusion. These files can be accessed using the following command:

tensorboard --logdir "/tb_logs"

Dataset Access

Access to the ‘IIT Roorkee Text and Image Emotion Recognition (IIT-R TIER) dataset’ can be obtained by through Access Form - IIT-R TIER Dataset.pdf. The dataset is compiled by Puneet Kumar, Yukti Gupta, and Vedanti Khokher at Machine Intelligence Lab, IIT Roorkee under the supervision of Prof. Balasubramanian Raman. It contains 97,170 images and corresponding text labeled with Emotion class, i.e., Happy, Sad, Hate, and Anger.