Towards in situ affect detection in mobile devices: a multimodal approach

Authors:
Mohammad Adibuzzaman;Niharika Jain;Nicholas Steinhafel;Munir Haque;Ferdaus Ahmed;Shiekh Iqbal Ahamed;Richard Love
Affiliations:
Marquette University, Milwaukee, WI;Marquette University, Milwaukee, WI;Marquette University, Milwaukee, WI;University of Alabama at Birmingham, Alabama;Marquette University, Milwaukee, WI;Marquette University, Milwaukee, WI;International Breast Cancer Research Foundation, Madison, WI
Venue:
Proceedings of the 2013 Research in Adaptive and Convergent Systems
Year:
2013

Citing 12
Cited 0

Some computer science issues in ubiquitous computing

Communications of the ACM - Special issue on computer augmented environments: back to the real world
Affective computing

Affective computing
Affective computing: challenges

International Journal of Human-Computer Studies - Application of affective computing in human—Computer interaction
Multimodal affect recognition in learning environments

Proceedings of the 13th annual ACM international conference on Multimedia
Multimodal human-computer interaction: A survey

Computer Vision and Image Understanding
A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions

IEEE Transactions on Pattern Analysis and Machine Intelligence
iPhone as a physical activity measurement platform

CHI '10 Extended Abstracts on Human Factors in Computing Systems
Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications

IEEE Transactions on Affective Computing
Continuous Prediction of Spontaneous Affect from Multiple Cues and Modalities in Valence-Arousal Space

IEEE Transactions on Affective Computing
Passive and In-Situ assessment of mental and physical well-being using mobile sensors

Proceedings of the 13th international conference on Ubiquitous computing
Recording affect in the field: towards methods and metrics for improving ground truth labels

ACII'11 Proceedings of the 4th international conference on Affective computing and intelligent interaction - Volume Part I
Out of the lab and into the fray: towards modeling emotion in everyday life

Pervasive'10 Proceedings of the 8th international conference on Pervasive Computing

Quantified Score

Hi-index	0.00

Visualization

Abstract

Most of the research in multi-modal affect detection has been done in laboratory environment. Little work has been done for in situ affect detection. In this paper, we investigate affect detection in natural environment using sensors available in smart phones. We use facial expression and energy expenditure of a person to classify a person's affective state by continuously capturing fine grained accelerometer data for energy and camera image for facial expression and measure the performance of the system. We have deployed our system in natural environment and have provided special attention on annotation for the training data validating the 'ground truth'. We have found important correlation between facial image and energy which validates Russell's two dimensional theory of emotion using arousal and valence space. In this paper, we have presented initial findings in multi-modal affect detection.