Analysis and synthesis of multimodal verbal and non-verbal interaction for animated interface agents

Authors:
Jonas Beskow;Björn Granström;David House
Affiliations:
Centre for Speech Technology, CSC, KTH, Stockholm, Sweden;Centre for Speech Technology, CSC, KTH, Stockholm, Sweden;Centre for Speech Technology, CSC, KTH, Stockholm, Sweden
Venue:
COST 2102'07 Proceedings of the 2007 COST action 2102 international conference on Verbal and nonverbal communication behaviours
Year:
2007

Citing 3
Cited 5

Human conversation as a system framework: designing embodied conversational agents

Embodied conversational agents
MPEG-4 Facial Animation: The Standard,Implementation and Applications

MPEG-4 Facial Animation: The Standard,Implementation and Applications
Parameterized Models for Facial Animation

IEEE Computer Graphics and Applications

SynFace: speech-driven facial animation for virtual speech-reading support

EURASIP Journal on Audio, Speech, and Music Processing - Special issue on animating virtual speakers or singers from audio: Lip-synching facial animation
Audiovisual alignment in a face-to-face conversation translation framework

BioID_MultiComm'09 Proceedings of the 2009 joint COST 2101 and 2102 international conference on Biometric ID management and multimodal communication
Face-to-face interaction and the KTH cooking show

COST'09 Proceedings of the Second international conference on Development of Multimodal Interfaces: active Listening and Synchrony
A phenomenological study of facial expression animation

BCS-HCI '11 Proceedings of the 25th BCS Conference on Human-Computer Interaction
Guest Editorial: Gesture and speech in interaction: An overview

Speech Communication

Quantified Score

Hi-index	0.00

Visualization

Abstract

The use of animated talking agents is a novel feature of many multimodal spoken dialogue systems. The addition and integration of a virtual talking head has direct implications for the way in which users approach and interact with such systems. However, understanding the interactions between visual expressions, dialogue functions and the acoustics of the corresponding speech presents a substantial challenge. Some of the visual articulation is closely related to the speech acoustics, while there are other articulatory movements affecting speech acoustics that are not visible on the outside of the face. Many facial gestures used for communicative purposes do not affect the acoustics directly, but might nevertheless be connected on a higher communicative level in which the timing of the gestures could play an important role. This chapter looks into the communicative function of the animated talking agent, and its effect on intelligibility and the flow of the dialogue.