Home

MMSA-Feature Extraction Tool

MMSA-Feature Extraction Tool extracts multimodal features for Multimodal Sentiment Analysis Datasets. It integrates several commonly used tools for visual, acoustic and text modality. The extracted features are compatible with the MMSA Framework and thus can be used directly. The tool can also extract features for single videos.

1. Installation

MMSA-Feature Extraction Tool is available from Pypi:

$ pip install MMSA-FET

For the OpenFaceExtractor to work, a few system-wide dependancies are needed. See Dependency Installation for more information.

2. Quick Start

MMSA-FET is very easy to use. Below is a basic example on how to extract features for a single video file and a dataset folder.

from MSA_FET import FeatureExtractionTool

# initialize with config file
fet = FeatureExtractionTool("config.json")

# extract features for single video
feature = fet.run_single("input.mp4")
print(feature)

# extract for dataset & save features to file
feature = fet.run_dataset(dataset_dir="~/MOSI", out_file="output/feature.pkl")

The config.json is the path to a custom config file, the format of which is introduced here

For more details, please read Usage.

3. Config File

The tool comes with a few example configs which can be used like below.

# Every tool has an example config
fet = FeatureExtractionTool(config="openface")
fet = FeatureExtractionTool(config="bert")
fet = FeatureExtractionTool(config="librosa")
...

For customized features, you'll have to provide a config file which is in the following format.

{
  "audio": {
    "tool": "librosa",
    "sample_rate": null,
    "args": {
      "mfcc": {
        "n_mfcc": 20,
        "htk": true
      },
      "rms": {},
      "zero_crossing_rate": {},
      "spectral_rolloff": {},
      "spectral_centroid": {}
    }
  },
  "video": {
    "tool": "openface",
    "fps": 25,
    "average_over": 3,
    "args": {
      "hogalign": false,
      "simalign": false,
      "nobadaligned": false,
      "landmark_2D": true,
      "landmark_3D": false,
      "pdmparams": false,
      "head_pose": true,
      "action_units": true,
      "gaze": true,
      "tracked": false
    }
  },
  "text": {
    "model": "bert",
    "device": "cpu",
    "pretrained": "models/bert_base_uncased",
    "args": {}
  }
}

4. Supported Tools & Features

4.1 Audio Tools

• Librosa (link)

  Supports all librosa features listed here, including: mfcc, rms, zero_crossing_rate, spectral_rolloff, spectral_centroid, etc.

• openSMILE (link)

  Supports all feature sets listed here, including: ComParE_2016, GeMAPS, eGeMAPS, emobase, etc.

• Wav2vec2 (link)

  Integrated from huggingface transformers.

4.2 Video Tools

• OpenFace

  Currently only supports Linux platform.