Emotion Recognition through Multiple Modalities: Face, Body Gesture, Speech

  • Authors:
  • Ginevra Castellano;Loic Kessous;George Caridakis

  • Affiliations:
  • InfoMus Lab, DIST - University of Genova, Genova, Italy I-16145;Department of Speech, Language and Hearing, University of Tel Aviv, Sheba Center, Tel Aviv, Israel 52621;Image, Video and Multimedia Systems Laboratory, National Technical University of Athens, Athens, Greece 15780

  • Venue:
  • Affect and Emotion in Human-Computer Interaction
  • Year:
  • 2008

Quantified Score

Hi-index 0.00

Visualization

Abstract

In this paper we present a multimodal approach for the recognition of eight emotions. Our approach integrates information from facial expressions, body movement and gestures and speech. We trained and tested a model with a Bayesian classifier, using a multimodal corpus with eight emotions and ten subjects. Firstly, individual classifiers were trained for each modality. Next, data were fused at the feature level and the decision level. Fusing the multimodal data resulted in a large increase in the recognition rates in comparison with the unimodal systems: the multimodal approach gave an improvement of more than 10% when compared to the most successful unimodal system. Further, the fusion performed at the feature level provided better results than the one performed at the decision level.