Skip to content

Latest commit

 

History

History
71 lines (51 loc) · 2.57 KB

README.md

File metadata and controls

71 lines (51 loc) · 2.57 KB
Raw

A Unimodal Valence-Arousal Driven Contrastive Learning Framework for Multimodal Multi-Label Emotion Recognition

Wenjie Zheng, Jianfei Yu, and Rui Xia

PyTorch Conference

📄 Paper 📽️ Slides

This repository contains the code for UniVA, a framework that proposes unimodal valence-arousal driven contrastive learning for the multimodal multi-label emotion recognition task.

overview

Dependencies

conda env create -f environment.yml

Data preparation

Download link. Also, there are two files that, due to upload size limitations, have been placed at the link.

Evaluating UniVA on the MOSEI dataset

you can check our UniVA-RoBERTa on 4 NVIDIA 3090 GPUs by running the script below

nohup bash run_MOSEI/run_MOSEI_TAV_ours.sh &

you can get the following results: Acc51.3 HL0.182 miF160.5 maF144.4

To evaluate the performance of UniVA-Glove on 1 NVIDIA 3090Ti GPU, run the script below

nohup bash run_MOSEI/run_MOSEI_TAV_ours_glove.sh &

you can get the following results: Acc49.2 HL0.205 miF157.2 maF137.2

Evaluating UniVA on the M3ED dataset

you can check our UniVA-RoBERTa on 4 NVIDIA 3090 GPUs by running the script below

nohup bash run_M3ED/run_M3ED_TAV_ours.sh &

you can get the following results: Acc50.6 HL0.149 miF153.4 maF140.2

To evaluate the performance of UniVA-Glove on 1 NVIDIA 3090Ti GPU, run the script below

nohup bash run_M3ED/run_M3ED_TAV_ours_glove.sh &

you can get the following results: Acc46.4 HL0.159 miF149.1 maF124.2

Citation

Please consider citing the following if this repo is helpful to your research.

@inproceedings{zheng2024unimodal,
  title={A Unimodal Valence-Arousal Driven Contrastive Learning Framework for Multimodal Multi-Label Emotion Recognition},
  author={Zheng, Wenjie and Yu, Jianfei and Xia, Rui},
  booktitle={Proceedings of the 32nd ACM International Conference on Multimedia},
  pages={622--631},
  year={2024}
}

Please let me know if I can future improve in this repositories or there is anything wrong in our work. You can ask questions via issues in Github or contact me via email [email protected]. Thanks for your support!