Smile and Laughter Recognition using Speech Processing and Face Recognition from Conversation Video

Authors:
Akinori Ito;Xinyue Wang;Motoyuki Suzuki;Shozo Makino
Affiliations:
Graduate School of Engineering, Tohoku University;Graduate School of Engineering, Tohoku University;Graduate School of Engineering, Tohoku University;Graduate School of Engineering, Tohoku University
Venue:
CW '05 Proceedings of the 2005 International Conference on Cyberworlds
Year:
2005

Citing 0
Cited 11

Fusion of audio and visual cues for laughter detection

CIVR '08 Proceedings of the 2008 international conference on Content-based image and video retrieval
Decision-Level Fusion for Audio-Visual Laughter Detection

MLMI '08 Proceedings of the 5th international workshop on Machine Learning for Multimodal Interaction
Audiovisual laughter detection based on temporal features

ICMI '08 Proceedings of the 10th international conference on Multimodal interfaces
Social signal processing: state-of-the-art and future perspectives of an emerging domain

MM '08 Proceedings of the 16th ACM international conference on Multimedia
Smile Detection for User Interfaces

ISVC '08 Proceedings of the 4th International Symposium on Advances in Visual Computing, Part II
Social signal processing: Survey of an emerging domain

Image and Vision Computing
Recognizing communicative facial expressions for discovering interpersonal emotions in group meetings

Proceedings of the 2009 international conference on Multimodal interfaces
A study for the self similarity smile detection

BioID_MultiComm'09 Proceedings of the 2009 joint COST 2101 and 2102 international conference on Biometric ID management and multimodal communication
Spotting laughter in natural multiparty conversations: A comparison of automatic online and offline approaches using audiovisual data

ACM Transactions on Interactive Intelligent Systems (TiiS) - Special Issue on Affective Interaction in Natural Environments
The MAHNOB Laughter database

Image and Vision Computing
Towards the automatic detection of spontaneous agreement and disagreement based on nonverbal behaviour: A survey of related cues, databases, and tools

Image and Vision Computing

Quantified Score

Hi-index	0.00

Visualization

Abstract

This paper describes a method to detect smiles and laughter sounds from the video of natural dialogue. A smile is the most common facial expression observed in a dialogue. Detecting a user's smiles and laughter sounds can be useful for estimating the mental state of the user of a spoken-dialogue-based user interface. In addition, detecting laughter sound can be utilized to prevent the speech recognizer from wrongly recognizing the laughter sound as meaningful words. In this paper, a method to detect smile expression and laughter sound robustly by combining an image-based facial expression recognition method and an audio-based laughter sound recognition method. The image-based method uses a feature vector based on feature point detection from face images. The method could detect smile faces by more than 80% recall and precision rate. A method to combine a GMM-based laughter sound recognizer and the image-based method could improve the accuracy of detection of laughter sounds compared with methods that use image or sound only. As a result, more than 70% recall and precision rate of laughter sound detection was obtained from the natural conversation videos.