Resolving Object References in Multimodal Dialogues for Immersive Virtual Environments

Authors:
Thies Pfeiffer;Marc Erich Latoschik
Affiliations:
University of Bielefeld, Germany;University of Bielefeld, Germany
Venue:
VR '04 Proceedings of the IEEE Virtual Reality 2004
Year:
2004

Citing 0
Cited 8

Agents That Talk And Hit Back: Animated Agents in Augmented Reality

ISMAR '04 Proceedings of the 3rd IEEE/ACM International Symposium on Mixed and Augmented Reality
A user interface framework for multimodal VR interactions

ICMI '05 Proceedings of the 7th international conference on Multimodal interfaces
Working with robots and objects: revisiting deictic reference for achieving spatial common ground

Proceedings of the 1st ACM SIGCHI/SIGART conference on Human-robot interaction
Introducing semantic information during conceptual modelling of interaction for virtual environments

Proceedings of the 2007 workshop on Multimodal interfaces in semantic interaction
VR-CAD integration: Multimodal immersive interaction and advanced haptic paradigms for implicit edition of CAD models

Computer-Aided Design
A multimodal reference resolution approach in virtual environment

VSMM'06 Proceedings of the 12th international conference on Interactive Technologies and Sociotechnical Systems
Deixis: how to determine demonstrated objects using a pointing cone

GW'05 Proceedings of the 6th international conference on Gesture in Human-Computer Interaction and Simulation
Knowledge in the loop: semantics representation for multimodal simulative environments

SG'05 Proceedings of the 5th international conference on Smart Graphics

Quantified Score

Hi-index	0.00

Visualization

Abstract

This paper describes the underlying concepts and thetechnical implementation of a system for resolving multi-modal references in Virtual Reality (VR). In this system thetemporal and semantic relations intrinsic to referential utterances are expressed as a constraint satisfaction problem,where the propositional value of each referential unit during a multimodal dialogue updates incrementally the active set of constraints. As the system is based on findings ofhuman cognition research it also regards, e.g., constraintsimplicitly assumed by human communicators. The implementation takes VR related real-time and immersive conditions into account and adapts its architecture to well knownscene-graph based design patterns by introducing a so-called reference resolution engine. Regarding the conceptual work as well as regarding the implementation, specialcare has been taken to allow further refinements and modifications to the underlying resolving processes on a high level basis.