Toward a decision-theoretic framework for affect recognition and user assistance

  • Authors:

  • Wenhui Liao;Weihong Zhang;Zhiwei Zhu;Qiang Ji;Wayne D. Gray

  • Affiliations:

  • Department of Electrical, Computer and Systems Engineering, Rensselaer Polytechnic Institute, Troy, NY 12180-3590, USA;Department of Electrical, Computer and Systems Engineering, Rensselaer Polytechnic Institute, Troy, NY 12180-3590, USA;Department of Electrical, Computer and Systems Engineering, Rensselaer Polytechnic Institute, Troy, NY 12180-3590, USA;Department of Electrical, Computer and Systems Engineering, Rensselaer Polytechnic Institute, Troy, NY 12180-3590, USA;Department of Cognitive Science, Rensselaer Polytechnic Institute, Troy, NY 12180-3590, USA

  • Venue:
  • International Journal of Human-Computer Studies - Human-computer interaction research in the managemant information systems discipline
  • Year:
  • 2006

Quantified Score

Hi-index 0.00

Visualization

Abstract

There is an increasing interest in developing intelligent human-computer interaction systems that can fulfill two functions-recognizing user affective states and providing the user with timely and appropriate assistance. In this paper, we present a general unified decision-theoretic framework based on influence diagrams for simultaneously modeling user affect recognition and assistance. Affective state recognition is achieved through active probabilistic inference from the available multi modality sensory data. User assistance is automatically accomplished through a decision-making process that balances the benefits of keeping the user in productive affective states and the costs of performing user assistance. We discuss three theoretical issues within the framework, namely, user affect recognition, active sensory action selection, and user assistance. Validation of the proposed framework via a simulation study demonstrates its capability in efficient user affect recognition as well as timely and appropriate user assistance. Besides the theoretical contributions, we build a non-invasive real-time prototype system to recognize different user affective states (stress and fatigue) from four-modality user measurements, namely physical appearance features, physiological measures, user performance, and behavioral data. The affect recognition component of the prototype system is subsequently validated through a real-world study involving human subjects.