Designing a multi-modal affective knowledge-based user interface: combining empirical studies

Authors:
Maria Virvou;George Tsihrintzis;Efthymios Alepis;Ioanna-Ourania Stathopoulou
Affiliations:
Department of Informatics, University of Piraeus, 80 Karaoli & Dimitriou St., 18534 Piraeus, Greece, E-mail: mvirvou@unipi.gr/ geoatsi@unipi.gr/ talepis@unipi.gr/ iostath@unipi.gr;Department of Informatics, University of Piraeus, 80 Karaoli & Dimitriou St., 18534 Piraeus, Greece, E-mail: mvirvou@unipi.gr/ geoatsi@unipi.gr/ talepis@unipi.gr/ iostath@unipi.gr;Department of Informatics, University of Piraeus, 80 Karaoli & Dimitriou St., 18534 Piraeus, Greece, E-mail: mvirvou@unipi.gr/ geoatsi@unipi.gr/ talepis@unipi.gr/ iostath@unipi.gr;Department of Informatics, University of Piraeus, 80 Karaoli & Dimitriou St., 18534 Piraeus, Greece, E-mail: mvirvou@unipi.gr/ geoatsi@unipi.gr/ talepis@unipi.gr/ iostath@unipi.gr
Venue:
Proceedings of the 2008 conference on Knowledge-Based Software Engineering: Proceedings of the Eighth Joint Conference on Knowledge-Based Software Engineering
Year:
2008

Citing 5
Cited 1

Multimodal Human Emotion/Expression Recognition

FG '98 Proceedings of the 3rd. International Conference on Face & Gesture Recognition
Affective computing: challenges

International Journal of Human-Computer Studies - Application of affective computing in human—Computer interaction
Visual Prosody: Facial Movements Accompanying Speech

FGR '02 Proceedings of the Fifth IEEE International Conference on Automatic Face and Gesture Recognition
Analysis of emotion recognition using facial expressions, speech and multimodal information

Proceedings of the 6th international conference on Multimodal interfaces
Affective Student Modeling Based on Microphone and Keyboard User Actions

ICALT '06 Proceedings of the Sixth IEEE International Conference on Advanced Learning Technologies

Visual affect recognition

Visual affect recognition

Quantified Score

Hi-index	0.00

Visualization

Abstract

In this paper, we present and discuss three empirical studies that we have conducted involving human subjects and human observers concerning the recognition of emotions from audio-lingual visual-facial and keyboard-evidence modalities. Many researchers agree that these modalities are complementary to each other and that their combination can improve the accuracy in affective user models. However, there is a shortage of research in empirical work concerning the strengths and weaknesses of each modality so that more accurate recognizers can be built. In our research, we have investigated the recognition of emotions with respect to 6 basic emotional states, namely happiness, sadness, surprise, anger and disgust as well as the emotionless state which we refer to as neutral. We have concluded that, in cases where a single modality may be deficient in providing emotion recognition evidence, the recognition process can be supported and complemented by the other modalities.