Animated conversation: rule-based generation of facial expression, gesture & spoken intonation for multiple conversational agents

Authors:
Justine Cassell;Catherine Pelachaud;Norman Badler;Mark Steedman;Brett Achorn;Tripp Becket;Brett Douville;Scott Prevost;Matthew Stone
Affiliations:
Department of Computer & Information Science, University of Pennsylvania;Department of Computer & Information Science, University of Pennsylvania;Department of Computer & Information Science, University of Pennsylvania;Department of Computer & Information Science, University of Pennsylvania;Department of Computer & Information Science, University of Pennsylvania;Department of Computer & Information Science, University of Pennsylvania;Department of Computer & Information Science, University of Pennsylvania;Department of Computer & Information Science, University of Pennsylvania;Department of Computer & Information Science, University of Pennsylvania
Venue:
SIGGRAPH '94 Proceedings of the 21st annual conference on Computer graphics and interactive techniques
Year:
1994

Citing 10
Cited 126

Simulation of object and human skin formations in a grasping task

SIGGRAPH '89 Proceedings of the 16th annual conference on Computer graphics and interactive techniques
Strength guided motion

SIGGRAPH '90 Proceedings of the 17th annual conference on Computer graphics and interactive techniques
Making them move: mechanics, control, and animation of articulated figures

Making them move: mechanics, control, and animation of articulated figures
Composition of realistic animation sequences for multiple human figures

Making them move
Human body deformations using joint-dependent local operators and finite-element theory

Making them move
Computer animation of knowledge-based human grasping

Proceedings of the 18th annual conference on Computer graphics and interactive techniques
Simulating humans: computer graphics animation and control

Simulating humans: computer graphics animation and control
Communicative facial displays as a new conversational modality

INTERCHI '93 Proceedings of the INTERCHI '93 conference on Human factors in computing systems
Computer graphic modeling of american sign language

SIGGRAPH '83 Proceedings of the 10th annual conference on Computer graphics and interactive techniques
Generating contextually appropriate intonation

EACL '93 Proceedings of the sixth conference on European chapter of the Association for Computational Linguistics

Behavioral control for real-time simulated human agents

I3D '95 Proceedings of the 1995 symposium on Interactive 3D graphics
Motion signal processing

SIGGRAPH '95 Proceedings of the 22nd annual conference on Computer graphics and interactive techniques
Providing assurances in a multimedia interactive environment

CHI '95 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Comic Chat

SIGGRAPH '96 Proceedings of the 23rd annual conference on Computer graphics and interactive techniques
A response model for a CG character based on timing of interactions in a multimodal human interface

Proceedings of the 2nd international conference on Intelligent user interfaces
Negotiation for automated generation of temporal multimedia presentations

MULTIMEDIA '96 Proceedings of the fourth ACM international conference on Multimedia
Integrating pedagogical capabilities in a virtual environment agent

AGENTS '97 Proceedings of the first international conference on Autonomous agents
Improvising linguistic style: social and affective bases for agent personality

AGENTS '97 Proceedings of the first international conference on Autonomous agents
Steve: an animated pedagogical agent for procedural training in virtual environments

ACM SIGART Bulletin
Modeling and generating sign language as animated line drawings

Assets '98 Proceedings of the third international ACM conference on Assistive technologies
Real-time decision making in multimodal face-to-face communication

AGENTS '98 Proceedings of the second international conference on Autonomous agents
BodyChat: autonomous communicative behaviors in avatars

AGENTS '98 Proceedings of the second international conference on Autonomous agents
Generating the animation of a 3D agent from explanation text

MULTIMEDIA '98 Proceedings of the sixth ACM international conference on Multimedia
Where to look? Automating attending behaviors of virtual human characters

Proceedings of the third annual conference on Autonomous Agents
Explanatory lifelike avatars: performing user-centered tasks in 3D learning environments

Proceedings of the third annual conference on Autonomous Agents
Embodiment in conversational interfaces: Rea

Proceedings of the SIGCHI conference on Human Factors in Computing Systems
Animation control for real-time virtual humans

Communications of the ACM
Ten myths of multimodal interaction

Communications of the ACM
More than just a pretty face: affordances of embodiment

Proceedings of the 5th international conference on Intelligent user interfaces
Verbal communication: using approximate sound propagation to design an inter-agents communication language

AGENTS '00 Proceedings of the fourth international conference on Autonomous agents
The EMOTE model for effort and shape

Proceedings of the 27th annual conference on Computer graphics and interactive techniques
BEAT: the Behavior Expression Animation Toolkit

Proceedings of the 28th annual conference on Computer graphics and interactive techniques
Speech-driven cartoon animation with emotions

MULTIMEDIA '01 Proceedings of the ninth ACM international conference on Multimedia
Gaze behavior of talking faces makes a difference

CHI '02 Extended Abstracts on Human Factors in Computing Systems
ACUMEN: amplifying control and understanding of multiple entities

Proceedings of the first international joint conference on Autonomous agents and multiagent systems: part 1
Embodied contextual agent in information delivering application

Proceedings of the first international joint conference on Autonomous agents and multiagent systems: part 2
Embodied agents for multi-party dialogue in immersive virtual worlds

Proceedings of the first international joint conference on Autonomous agents and multiagent systems: part 2
Leashing the AlphaWolves: mixing user direction with autonomous emotion in a pack of semi-autonomous virtual characters

Proceedings of the 2002 ACM SIGGRAPH/Eurographics symposium on Computer animation
Eyes alive

Proceedings of the 29th annual conference on Computer graphics and interactive techniques
A multilayer personality model

Proceedings of the 2nd international symposium on Smart graphics
Recognition of Affective Communicative Intent in Robot-Directed Speech

Autonomous Robots
Lifelike Pedagogical Agents for Mixed-initiative Problem Solving in Constructivist Learning Environments

User Modeling and User-Adapted Interaction
Lessons learned in modeling schizophrenic and depressed responsive virtual humans for training

Proceedings of the 8th international conference on Intelligent user interfaces
User interfaces and help systems: from helplessness to intelligent assistance

Artificial Intelligence Review
Fully Embodied Conversational Avatars: Making Communicative Behaviors Autonomous

Autonomous Agents and Multi-Agent Systems
Where to Look? Automating Attending Behaviors of Virtual Human Characters

Autonomous Agents and Multi-Agent Systems
Anthropoid Agents

IEEE Internet Computing
Automating Virtual Reality

IEEE Computer Graphics and Applications
Design of a Virtual Human Presenter

IEEE Computer Graphics and Applications
Face-to-face interfaces

CHI '99 Extended Abstracts on Human Factors in Computing Systems
Viewpoint adaptation during navigation based on stimuli from the virtual environment

Web3D '03 Proceedings of the eighth international conference on 3D Web technology
From Simulated Dialogues to Interactive Performances

Proceedings of the 9th ECCAI-ACAI/EASSS 2001, AEMAS 2001, HoloMAS 2001 on Multi-Agent-Systems and Applications II-Selected Revised Papers
Active Character: Dynamic Reaction to the User

GW '99 Proceedings of the International Gesture Workshop on Gesture-Based Communication in Human-Computer Interaction
A Complete System for the Specification and the Generation of Sign Language Gestures

GW '99 Proceedings of the International Gesture Workshop on Gesture-Based Communication in Human-Computer Interaction
A Sound Propagation Model for Interagents Communication

VW '00 Proceedings of the Second International Conference on Virtual Worlds
Intelligent Virtual Agents for Education and Training: Opportunities and Challenges

IVA '01 Proceedings of the Third International Workshop on Intelligent Virtual Agents
Emotion Detection from Speech to Enrich Multimedia Content

PCM '01 Proceedings of the Second IEEE Pacific Rim Conference on Multimedia: Advances in Multimedia Information Processing
Visages parlants: quels visages pour quelles voix ?

IHM '02 Proceedings of the 14th French-speaking conference on Human-computer interaction (Conférence Francophone sur l'Interaction Homme-Machine)
Extending virtual humans to support team training in virtual reality

Exploring artificial intelligence in the new millennium
Creating animations using virtual reality ThatcherWorld: a case study

CA '95 Proceedings of the Computer Animation
What Governs Autonomous Actors

CA '95 Proceedings of the Computer Animation
Motion Generator Approach to Translating Human Motion from Video to Animation

PG '99 Proceedings of the 7th Pacific Conference on Computer Graphics and Applications
Talking Heads: Which Matching between Faces and Synthetic Voices?

ICMI '02 Proceedings of the 4th IEEE International Conference on Multimodal Interfaces
Simulation of face-to-face interaction

AVI '96 Proceedings of the workshop on Advanced visual interfaces
Multimodal communication between synthetic agents

AVI '98 Proceedings of the working conference on Advanced visual interfaces
Agent-oriented simulation

Applied system simulation
Speaking with hands: creating animated conversational characters from recordings of human performance

ACM SIGGRAPH 2004 Papers
Internet Communication Using Real-Time Facial Expression Analysis and Synthesis

IEEE MultiMedia
Audio-based head motion synthesis for Avatar-based telepresence systems

Proceedings of the 2004 ACM SIGMM workshop on Effective telepresence
Real-time speech motion synthesis from recorded motions

SCA '04 Proceedings of the 2004 ACM SIGGRAPH/Eurographics symposium on Computer animation
A virtual human architecture that integrates kinematic, physical and behavioral aspects to control H-Anim characters

Web3D '05 Proceedings of the tenth international conference on 3D Web technology
Automated Eye Motion Using Texture Synthesis

IEEE Computer Graphics and Applications
Creating Speech-Synchronized Animation

IEEE Transactions on Visualization and Computer Graphics
Mood swings: expressive speech animation

ACM Transactions on Graphics (TOG)
Automatic determination of facial muscle activations from sparse motion capture marker data

ACM SIGGRAPH 2005 Papers
ALMA: a layered model of affect

Proceedings of the fourth international joint conference on Autonomous agents and multiagent systems
Expressive speech-driven facial animation

ACM Transactions on Graphics (TOG)
Direction of attention perception for conversation initiation in virtual environments

Lecture Notes in Computer Science
Coordination and context-dependence in the generation of embodied conversation

INLG '00 Proceedings of the first international conference on Natural language generation - Volume 14
Design of animated pedagogical agents-A look at their look

International Journal of Human-Computer Studies
Adaptive decision support system (ADSS) for B2C e-commerce

ICEC '06 Proceedings of the 8th international conference on Electronic commerce: The new e-commerce: innovations for conquering current barriers, obstacles and limitations to conducting successful business on the internet
Affective affordances: Improving interface character engagement through interaction

International Journal of Human-Computer Studies - Human-computer interaction research in the managemant information systems discipline
Expressive Facial Animation Synthesis by Learning Speech Coarticulation and Expression Spaces

IEEE Transactions on Visualization and Computer Graphics
HMM-based synthesis of emotional facial expressions during speech in synthetic talking heads

Proceedings of the 8th international conference on Multimodal interfaces
Building active conversation environment for edutainment

SIGGRAPH '05 ACM SIGGRAPH 2005 Posters
Simulating speech with a physics-based facial muscle model

Proceedings of the 2006 ACM SIGGRAPH/Eurographics symposium on Computer animation
Realism is not all! User engagement with task-related interface characters

Interacting with Computers
Employing virtual humans for education and training in X3D/VRML worlds

Computers & Education
Assembling an expressive facial animation system

Proceedings of the 2007 ACM SIGGRAPH symposium on Video games
Multispace behavioral model for face-based affective social agents

Journal on Image and Video Processing
A framework for socially communicative faces for game and interactive learning applications

Future Play '07 Proceedings of the 2007 conference on Future Play
Gesture modeling and animation based on a probabilistic re-creation of speaker style

ACM Transactions on Graphics (TOG)
Towards Natural Gesture Synthesis: Evaluating Gesture Units in a Data-Driven Approach to Gesture Synthesis

IVA '07 Proceedings of the 7th international conference on Intelligent Virtual Agents
Animating Speech in Games

Motion in Games
Mutually Coordinated Anticipatory Multimodal Interaction

Verbal and Nonverbal Features of Human-Human and Human-Machine Interaction
Footing in human-robot conversations: how robots might shape participant roles using gaze cues

Proceedings of the 4th ACM/IEEE international conference on Human robot interaction
Towards Facial Gestures Generation by Speech Signal Analysis Using HUGE Architecture

Multimodal Signals: Cognitive and Algorithmic Issues
Converting text into agent animations: assigning gestures to text

HLT-NAACL-Short '04 Proceedings of HLT-NAACL 2004: Short Papers
Real-time prosody-driven synthesis of body language

ACM SIGGRAPH Asia 2009 papers
Towards generation of fluent referring action in multimodal situations

ReferringPhenomena '97 Referring Phenomena in a Multimedia Context and their Computational Treatment
Planning referential acts for animated presentation agents

ReferringPhenomena '97 Referring Phenomena in a Multimedia Context and their Computational Treatment
Behavior planning for a reflexive agent

IJCAI'01 Proceedings of the 17th international joint conference on Artificial intelligence - Volume 2
Investigating effective ECAs: an experiment on modality and initiative

INTERACT'07 Proceedings of the 11th IFIP TC 13 international conference on Human-computer interaction - Volume Part II
Gesture controllers

ACM SIGGRAPH 2010 papers
Affective negotiation support systems

Journal of Ambient Intelligence and Smart Environments
The perception of finger motions

Proceedings of the 7th Symposium on Applied Perception in Graphics and Visualization
Uncertainty in Spoken Dialogue Management

Proceedings of the 2010 conference on Human Language Technologies -- The Baltic Perspective: Proceedings of the Fourth International Conference Baltic HLT 2010
Audio-visual prosody: perception, detection, and synthesis of prominence

Proceedings of the Third COST 2102 international training school conference on Toward autonomous, adaptive, and context-aware multimodal interfaces: theoretical and practical issues
Improving performance and retention in computer science courses using a virtual game show

Proceedings of the 49th Annual Southeast Regional Conference
Head movements, facial expressions and feedback in Danish first encounters interactions: a culture-specific analysis

UAHCI'11 Proceedings of the 6th international conference on Universal access in human-computer interaction: users diversity - Volume Part II
Interdisciplinary contributions to flame modeling

AI*IA'11 Proceedings of the 12th international conference on Artificial intelligence around man and beyond
Evaluating importance of facial expression in american sign language and pidgin signed english animations

The proceedings of the 13th international ACM SIGACCESS conference on Computers and accessibility
Conversational gaze mechanisms for humanlike robots

ACM Transactions on Interactive Intelligent Systems (TiiS)
ComPLuS model: A new insight in pupils' collaborative talk, actions and balance during a computer-mediated music task

Computers & Education
Perceiving visual emotions with speech

IVA'06 Proceedings of the 6th international conference on Intelligent Virtual Agents
Towards a common framework for multimodal generation: the behavior markup language

IVA'06 Proceedings of the 6th international conference on Intelligent Virtual Agents
Creativity meets automation: combining nonverbal action authoring with rules and machine learning

IVA'06 Proceedings of the 6th international conference on Intelligent Virtual Agents
[HUGE]: universal architecture for statistically based HUman GEsturing

IVA'06 Proceedings of the 6th international conference on Intelligent Virtual Agents
A story about gesticulation expression

IVA'06 Proceedings of the 6th international conference on Intelligent Virtual Agents
Synthesizing mood-affected signed messages: Modifications to the parametric synthesis

International Journal of Human-Computer Studies
Locomotion control technique for immersive conversation environment

KES'05 Proceedings of the 9th international conference on Knowledge-Based Intelligent Information and Engineering Systems - Volume Part III
Examining the role of gestures in expert tutoring

ITS'10 Proceedings of the 10th international conference on Intelligent Tutoring Systems - Volume Part I
Sleight of hand: perception of finger motion from reduced marker sets

I3D '12 Proceedings of the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games
Making robots persuasive: the influence of combining persuasive strategies (gazing and gestures) by a storytelling robot on its persuasive power

ICSR'11 Proceedings of the Third international conference on Social Robotics
Virtual humans: back to the future

Proceedings of Graphics Interface 2012
Emotional eye movement generation based on Geneva Emotion Wheel for virtual agents

Journal of Visual Languages and Computing
Towards building a virtual counselor: modeling nonverbal behavior during intimate self-disclosure

Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems - Volume 1
Advanced authoring tools for game-based training

SCSC '09 Proceedings of the 2009 Summer Computer Simulation Conference
An NVC emotional model for conversational virtual humans in a 3d chatting environment

AMDO'12 Proceedings of the 7th international conference on Articulated Motion and Deformable Objects
Improving gestural communication in virtual characters

AMDO'12 Proceedings of the 7th international conference on Articulated Motion and Deformable Objects
An interdisciplinary VR-architecture for 3D chatting with non-verbal communication

EGVE - JVRC'11 Proceedings of the 17th Eurographics conference on Virtual Environments & Third Joint Virtual Reality
3D animated agent for tutoring based on WebGL

Proceedings of the 18th International Conference on 3D Web Technology
Virtual character performance from speech

Proceedings of the 12th ACM SIGGRAPH/Eurographics Symposium on Computer Animation
Consideration of the effect of gesture exaggeration in web3D communication using 3DAgent

HCI'13 Proceedings of the 15th international conference on Human Interface and the Management of Information: information and interaction for learning, culture, collaboration and business - Volume Part III
Vers des Agents Conversationnels Animés Socio-Affectifs

Proceedings of the 25ième conférence francophone on l'Interaction Homme-Machine
Gesture synthesis adapted to speech emphasis

Speech Communication

Quantified Score

Hi-index	0.04

Visualization

Abstract

We describe an implemented system which automatically generates and animates conversations between multiple human-like agents with appropriate and synchronized speech, intonation, facial expressions, and hand gestures. Conversation is created by a dialogue planner that produces the text as well as the intonation of the utterances. The speaker/listener relationship, the text, and the intonation in turn drive facial expressions, lip motions, eye gaze, head motion, and arm gestures generators. Coordinated arm, wrist, and hand motions are invoked to create semantically meaningful gestures. Throughout we will use examples from an actual synthesized, fully animated conversation.