The hinge between input and output: understanding the multimodal input fusion results in an agent-based multimodal presentation system

Authors:
Yong Sun;Helmut Prendinger;Yu (David) Shi;Fang Chen;Vera Chung;Mitsuru Ishizuka
Affiliations:
National ICT Australia, Sydney, Australia;National Institute of Informatics, Japan, Tokyo, Japan;National ICT Australia, Sydney, Australia;National ICT Australia, Sydney, Australia;The University of Sydney, Australia, Sydney, Australia;University of Tokyo, Tokyo, Japan
Venue:
CHI '08 Extended Abstracts on Human Factors in Computing Systems
Year:
2008

Citing 8
Cited 2

Implementation and evaluation of a constraint-based multimodal fusion system for speech and 3D pointing gestures

Proceedings of the 6th international conference on Multimodal interfaces
Finite-state multimodal integration and understanding

Natural Language Engineering
Coupling CCG and hybrid logic dependency semantics

ACL '02 Proceedings of the 40th Annual Meeting on Association for Computational Linguistics
Put a grammar here: bi-directional parsing in multimodal interaction

CHI '06 Extended Abstracts on Human Factors in Computing Systems
Intelligent Interactive Entertainment Grand Challenges

IEEE Intelligent Systems
An Efficient Multimodal Language Processor for Parallel Input Strings in Multimodal Input Fusion

ICSC '07 Proceedings of the International Conference on Semantic Computing
An efficient unification-based multimodal language processor in multimodal input fusion

OZCHI '07 Proceedings of the 19th Australasian conference on Computer-Human Interaction: Entertaining User Interfaces
MPML3D: a reactive framework for the multimodal presentation markup language

IVA'06 Proceedings of the 6th international conference on Intelligent Virtual Agents

Skipping spare information in multimodal inputs during multimodal input fusion

Proceedings of the 14th international conference on Intelligent user interfaces
Building a Practical Multimodal System with a Multimodal Fusion Module

Proceedings of the 13th International Conference on Human-Computer Interaction. Part II: Novel Interaction Methods and Techniques

Quantified Score

Hi-index	0.00

Visualization

Abstract

A multimodal interface provides multiple modalities for input and output, such as speech, eye gaze and facial expression. With the recent progresses in multimodal interfaces, various approaches about multimodal input fusion and output generation have been proposed. However, less attention has been paid to how to integrate them together in a multimodal input and output system. This paper proposes an approach, termed as THE HINGE, in providing agent-based multimodal presentations in accordance with multimodal input fusion results. The analysis of experiment result shows the proposed approach enhances the flexibility of the system while maintains its stability.