Fusing face and body display for bi-modal emotion recognition: single frame analysis and multi-frame post integration

Authors:
Hatice Gunes;Massimo Piccardi
Affiliations:
Faculty of Information Technology, University of Technology, Sydney (UTS), Broadway, NSW, Australia;Faculty of Information Technology, University of Technology, Sydney (UTS), Broadway, NSW, Australia
Venue:
ACII'05 Proceedings of the First international conference on Affective Computing and Intelligent Interaction
Year:
2005

Citing 9
Cited 6

Computer vision and image processing

Computer vision and image processing
Data mining: practical machine learning tools and techniques with Java implementations

Data mining: practical machine learning tools and techniques with Java implementations
A Theoretical Study on Six Classifier Fusion Strategies

IEEE Transactions on Pattern Analysis and Machine Intelligence
Bimodal Emotion Recognition

FG '00 Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition 2000
To feel or not to feel: the role of affect in human-computer interaction

International Journal of Human-Computer Studies - Application of affective computing in human—Computer interaction
Probabilistic Combination of Multiple Modalities to Detect Interest

ICPR '04 Proceedings of the Pattern Recognition, 17th International Conference on (ICPR'04) Volume 3 - Volume 03
An iterative image registration technique with an application to stereo vision

IJCAI'81 Proceedings of the 7th international joint conference on Artificial intelligence - Volume 2
Emotion analysis in man-machine interaction systems

MLMI'04 Proceedings of the First international conference on Machine Learning for Multimodal Interaction
Multimodal integration-a statistical view

IEEE Transactions on Multimedia

Observer annotation of affective display and evaluation of expressivity: face vs. face-and-body

VisHCI '06 Proceedings of the HCSNet workshop on Use of vision in human-computer interaction - Volume 56
Decision-Level Fusion for Audio-Visual Laughter Detection

MLMI '08 Proceedings of the 5th international workshop on Machine Learning for Multimodal Interaction
Manual annotation and automatic image processing of multimodal emotional behaviors: validating the annotation of TV interviews

Personal and Ubiquitous Computing
Front view vs. side view of facial and postural expressions of emotions in a virtual character

Transactions on edutainment VI
Consistent but modest: a meta-analysis on unimodal and multimodal affect detection accuracies from 30 studies

Proceedings of the 14th ACM international conference on Multimodal interaction
A multimodal approach for online estimation of subtle facial expression

PCM'12 Proceedings of the 13th Pacific-Rim conference on Advances in Multimedia Information Processing

Quantified Score

Hi-index	0.00

Visualization

Abstract

This paper presents an approach to automatic visual emotion recognition from two modalities: expressive face and body gesture. Face and body movements are captured simultaneously using two separate cameras. For each face and body image sequence single “expressive” frames are selected manually for analysis and recognition of emotions. Firstly, individual classifiers are trained from individual modalities for mono-modal emotion recognition. Secondly, we fuse facial expression and affective body gesture information at the feature and at the decision-level. In the experiments performed, the emotion classification using the two modalities achieved a better recognition accuracy outperforming the classification using the individual facial modality. We further extend the affect analysis into a whole image sequence by a multi-frame post integration approach over the single frame recognition results. In our experiments, the post integration based on the fusion of face and body has shown to be more accurate than the post integration based on the facial modality only.