User and discourse models for multimodal communication
Intelligent user interfaces
QuickSet: multimodal interaction for distributed applications
MULTIMEDIA '97 Proceedings of the fifth ACM international conference on Multimedia
Ten myths of multimodal interaction
Communications of the ACM
Toward a theory of organized multimodal integration patterns during human-computer interaction
Proceedings of the 5th international conference on Multimodal interfaces
Unification-based multimodal integration
ACL '98 Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics and Eighth Conference of the European Chapter of the Association for Computational Linguistics
MULTIPLATFORM testbed: an integration platform for multimodal dialog systems
SEALTS '03 Proceedings of the HLT-NAACL 2003 workshop on Software engineering and architecture of language technology systems - Volume 8
Event-based multimedia chronicling systems
CARPE '05 Proceedings of the 2nd ACM workshop on Continuous archival and retrieval of personal experiences
From vocal to multimodal dialogue management
Proceedings of the 8th international conference on Multimodal interfaces
The SAMMIE system: multimodal in-car dialogue
COLING-ACL '06 Proceedings of the COLING/ACL on Interactive presentation sessions
A pen and speech-based storytelling system for Chinese children
Computers in Human Behavior
Natural and Intuitive Multimodal Dialogue for In-Car Applications: The Sammie System
Proceedings of the 2006 conference on ECAI 2006: 17th European Conference on Artificial Intelligence August 29 -- September 1, 2006, Riva del Garda, Italy
A Story Authoring System for Children
Edutainment '09 Proceedings of the 4th International Conference on E-Learning and Games: Learning by Playing. Game-based Education System Design and Development
Fusion engines for multimodal input: a survey
Proceedings of the 2009 international conference on Multimodal interfaces
A fusion framework for multimodal interactive applications
Proceedings of the 2009 international conference on Multimodal interfaces
Proceedings of the 2009 international conference on Multimodal interfaces
Generation of output style variation in the SAMMIE dialogue system
INLG '08 Proceedings of the Fifth International Natural Language Generation Conference
A multimodal 3D storytelling system for Chinese children
Edutainment'07 Proceedings of the 2nd international conference on Technologies for e-learning and digital entertainment
A comprehensive context model for multi-party interactions with virtual characters
IVA'06 Proceedings of the 6th international conference on Intelligent Virtual Agents
A multimodal fusion framework for children's storytelling systems
Edutainment'06 Proceedings of the First international conference on Technologies for E-Learning and Digital Entertainment
Using the transferable belief model for multimodal input fusion in companion systems
MPRSS'12 Proceedings of the First international conference on Multimodal Pattern Recognition of Social Signals in Human-Computer-Interaction
| Hi-index | 0.00 |
We present a generic approach to multimodal fusion which we call context based multimodal integration. Key to this approach is that every multimodal input event is interpreted and enriched with respect to its local turn context. This local turn context comprises all previously recognized input events and the dialogue state that both belong to the same user turn. We show that a production rule system is an elegant way to handle this context based multimodal integration and we describe a first implementation of the so-called PATE system. Finally, we present results from a first evaluation of this approach as part of a human-factors experiment with the COMIC system.