Cross-modal coordination of expressive strength between voice and gesture for personified media

Authors:
Tomoko Yonezawa;Noriko Suzuki;Shinji Abe;Kenji Mase;Kiyoshi Kogure
Affiliations:
ATR-IRC Labs. and Nagoya University, Soraku, Kyoto, Japan;ATR-IRC Labs, Soraku, Kyoto, Japan;ATR-IRC Labs, Soraku, Kyoto, Japan;Nagoya University and ATR-IRC Labs, Furo-Cho, Nagoya, Japan;ATR-MIS Labs, Soraku, Kyoto, Japan
Venue:
Proceedings of the 8th international conference on Multimodal interfaces
Year:
2006

Citing 7
Cited 2

The role of emotion in believable agents

Communications of the ACM
An open architecture for robot entertainment

AGENTS '97 Proceedings of the first international conference on Autonomous agents
Restructuring speech representations using a pitch-adaptive time-frequency smoothing and an instantaneous-frequency-based F0 extraction: possible role of a repetitive structure in sounds

Speech Communication
Embodied conversational interface agents

Communications of the ACM
RobotPHONE: RUI for interpersonal communication

CHI '01 Extended Abstracts on Human Factors in Computing Systems
Musically Expressive Doll in Face-to-Face Communication

ICMI '02 Proceedings of the 4th IEEE International Conference on Multimodal Interfaces
HandySinger: expressive singing voice morphing using personified hand-puppet interface

NIME '05 Proceedings of the 2005 conference on New interfaces for musical expression

Gaze-communicative behavior of stuffed-toy robot with joint attention and eye contact based on ambient gaze-tracking

Proceedings of the 9th international conference on Multimodal interfaces
The "one-person choir": A multidisciplinary approach to the development of an embodied human-computer interface

Computer Music Journal

Quantified Score

Hi-index	0.00

Visualization

Abstract

The aim of this paper is to clarify the relationship between the expressive strengths of gestures and voice for embodied and personified interfaces. We conduct perceptual tests using a puppet interface, while controlling singing-voice expressions, to empirically determine the naturalness and strength of various combinations of gesture and voice. The results show that (1) the strength of cross-modal perception is affected more by gestural expression than by the expressions of a singing voice, and (2) the appropriateness of cross-modal perception is affected by expressive combinations between singing voice and gestures in personified expressions. As a promising solution, we propose balancing a singing voice and gestural expressions by expanding and correcting the width and shape of the curve of expressive strength in the singing voice.