Avatar Puppetry Using Real-Time Audio and Video Analysis

Authors:
Sylvain Gallou;Gaspard Breton;Renaud Séguier;Christophe Garcia
Affiliations:
France Telecom, TECH/IRIS Team, rue du Clos Courtel, Cesson-Sévigné, France and Supélec/IETR, SCEE Team, avenue de la Boulaie, Cesson-Sévigné, France;France Telecom, TECH/IRIS Team, rue du Clos Courtel, Cesson-Sévigné, France;Supélec/IETR, SCEE Team, avenue de la Boulaie, Cesson-Sévigné, France;France Telecom, TECH/IRIS Team, rue du Clos Courtel, Cesson-Sévigné, France
Venue:
IVA '07 Proceedings of the 7th international conference on Intelligent Virtual Agents
Year:
2007

Citing 3
Cited 1

FaceEngine a 3D facial animation engine for real time applications

Proceedings of the sixth international conference on 3D Web technology
Active Appearance Models

ECCV '98 Proceedings of the 5th European Conference on Computer Vision-Volume II - Volume II
Modeling gaze behavior for a 3D ECA in a dialogue situation

Proceedings of the 11th international conference on Intelligent user interfaces

Control of speech-related facial movements of an avatar from video

Speech Communication

Quantified Score

Hi-index	0.00

Visualization

Abstract

We present a system which consists of a lifelike agent animated in real-time using video and audio analysis from the user. This kind of system could be used for Instant Messaging where an avatar controlled like a puppet is displayed instead of the webcam flow. The overall system is made of video analysis based on Active Appearance Models and audio analysis based on Hidden Markov Model. The parameters from these two modules are sent to a control system driving the animation engine. The video analysis extracts the head orientation and the audio analysis provides the phonetic string used to move the lips.