Combining semantic and temporal constraints for multimodal integration in conversation systems

Authors:
Joyce Y. Chai;Pengyu Hong;Michelle X. Zhou
Affiliations:
Michigan State University, East Lansing, MI;Harvard University, Cambridge, MA;IBM T. J. Watson Research Center, Hawthorne, NY
Venue:
HLT-NAACL-DIALOGUE '03 Proceedings of the HLT-NAACL 2003 workshop on Research directions in dialogue processing - Volume 7
Year:
2003

Citing 6
Cited 0

Automatic referent resolution of deictic and anaphoric expressions

Computational Linguistics
A Graduated Assignment Algorithm for Graph Matching

IEEE Transactions on Pattern Analysis and Machine Intelligence
QuickSet: multimodal interaction for distributed applications

MULTIMEDIA '97 Proceedings of the fifth ACM international conference on Multimedia
Natural language with integrated deictic and graphic gestures

Readings in intelligent user interfaces
User and discourse models for multimodal communication

Readings in intelligent user interfaces
Cognitive Status and Form of Reference in Multimodal Human-Computer Interaction

Proceedings of the Seventeenth National Conference on Artificial Intelligence and Twelfth Conference on Innovative Applications of Artificial Intelligence

Quantified Score

Hi-index	0.00

Visualization

Abstract

In a multimodal conversation, user referring patterns could be complex, involving multiple referring expressions from speech utterances and multiple gestures. To resolve those references, multimodal integration based on semantic constraints is insufficient. In this paper, we describe a graph-based probabilistic approach that simultaneously combines both semantic and temporal constraints to achieve a high performance.