Implementation and evaluation of a multimodal addressee identification mechanism for multiparty conversation systems

Authors:
Yukiko I. Nakano;Naoya Baba;Hung-Hsuan Huang;Yuki Hayashi
Affiliations:
Seikei University, Tokyo, Japan;Seikei University, Tokyo, Japan;Ritsumeikan University, Shiga, Japan;Seikei University, Tokyo, Japan
Venue:
Proceedings of the 15th ACM on International conference on multimodal interaction
Year:
2013

Citing 15
Cited 0

Eye gaze patterns in conversations: there is more to conversational agents than meets the eyes

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Video cut editing rule based on participants' gaze in multiparty conversation

MULTIMEDIA '03 Proceedings of the eleventh ACM international conference on Multimedia
Identifying the addressee in human-human-robot interactions based on head pose and speech

Proceedings of the 6th international conference on Multimodal interfaces
Identifying the intended addressee in mixed human-human and human-computer interaction from non-verbal features

ICMI '05 Proceedings of the 7th international conference on Multimodal interfaces
Human perception of intended addressee during computer-assisted meetings

Proceedings of the 8th international conference on Multimodal interfaces
Toward open-microphone engagement for multiparty interactions

Proceedings of the 8th international conference on Multimodal interfaces
Multimodalcues for addressee-hood in triadic communication with a human information retrieval agent

Proceedings of the 9th international conference on Multimodal interfaces
The design of a generic framework for integrating ECA components

Proceedings of the 7th international joint conference on Autonomous agents and multiagent systems - Volume 1
Multi-party, Multi-issue, Multi-strategy Negotiation for Multi-modal Virtual Agents

IVA '08 Proceedings of the 8th international conference on Intelligent Virtual Agents
Multimodal floor control shift detection

Proceedings of the 2009 international conference on Multimodal interfaces
Dialog in the open world: platform and applications

Proceedings of the 2009 international conference on Multimodal interfaces
Facilitating multiparty dialog with gaze, gesture, and speech

International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction
Making virtual conversational agent aware of the addressee of users' utterances in multi-user conversation using nonverbal information

ICMI '11 Proceedings of the 13th international conference on multimodal interfaces
Multiparty turn taking in situated dialog: study, lessons, and directions

SIGDIAL '11 Proceedings of the SIGDIAL 2011 Conference
Learning speaker, addressee and overlap detection models from multimodal streams

Proceedings of the 14th ACM international conference on Multimodal interaction

Quantified Score

Hi-index	0.00

Visualization

Abstract

In conversational agents with multiparty communication functionality, a system needs to be able to identify the addressee for the current floor and respond to the user when the utterance is addressed to the agent. This study proposes some addressee identification models based on speech and gaze information, and tests whether the models can be applied to different proxemics. We build an addressee identification mechanism by implementing the models and incorporate it into a fully autonomous multiparty conversational agent. The system identifies the addressee from online multimodal data and uses this information in language understanding and dialogue management. Finally, an evaluation experiment shows that the proposed addressee identification mechanism works well in a real-time system, with an F-measure for addressee estimation of 0.8 for agent-addressed utterances. We also found that our system more successfully avoided disturbing the conversation by mistakenly taking a turn when the agent is not addressed.