A multimodal reference resolution approach in virtual environment

Authors:
Xiaowu Chen;Nan Xu
Affiliations:
The Key Laboratory of Virtual Reality Technology, Ministry of Education, China;The Key Laboratory of Virtual Reality Technology, Ministry of Education, China
Venue:
VSMM'06 Proceedings of the 12th international conference on Interactive Technologies and Sociotechnical Systems
Year:
2006

Citing 17
Cited 0

Integrating simultaneous input from speech, gaze, and hand gestures

Intelligent multimedia interfaces
QuickSet: multimodal interaction for distributed applications

MULTIMEDIA '97 Proceedings of the fifth ACM international conference on Multimedia
Ten myths of multimodal interaction

Communications of the ACM
Cognitive Status and Form of Reference in Multimodal Human-Computer Interaction

Proceedings of the Seventeenth National Conference on Artificial Intelligence and Twelfth Conference on Innovative Applications of Artificial Intelligence
“Put-that-there”: Voice and gesture at the graphics interface

SIGGRAPH '80 Proceedings of the 7th annual conference on Computer graphics and interactive techniques
The Reality of Virtual Reality

VSMM '01 Proceedings of the Seventh International Conference on Virtual Systems and Multimedia (VSMM'01)
SenseShapes: Using Statistical Geometry for Object Selection in a Multimodal Augmented Reality System

ISMAR '03 Proceedings of the 2nd IEEE/ACM International Symposium on Mixed and Augmented Reality
Mutual disambiguation of 3D multimodal interaction in augmented and virtual reality

Proceedings of the 5th international conference on Multimodal interfaces
A probabilistic approach to reference resolution in multimodal user interfaces

Proceedings of the 9th international conference on Intelligent user interfaces
A model for multimodal reference resolution

Computational Linguistics
Finite-state multimodal parsing and understanding

COLING '00 Proceedings of the 18th conference on Computational linguistics - Volume 1
Resolving Object References in Multimodal Dialogues for Immersive Virtual Environments

VR '04 Proceedings of the IEEE Virtual Reality 2004
Linguistic theories in efficient multimodal reference resolution: an empirical investigation

Proceedings of the 10th international conference on Intelligent user interfaces
Semantics-based representation for multimodal interpretation in conversational systems

COLING '02 Proceedings of the 19th international conference on Computational linguistics - Volume 1
A user interface framework for multimodal VR interactions

ICMI '05 Proceedings of the 7th international conference on Multimodal interfaces
A Virtual Environment for Collaborative Assembly

ICESS '05 Proceedings of the Second International Conference on Embedded Software and Systems
Multimodal interactive maps: designing for human performance

Human-Computer Interaction

Quantified Score

Hi-index	0.00

Visualization

Abstract

This paper presents a multimodal reference resolution approach in virtual environment, which is called RRVE. Based on the relationship between cognitive status and reference, RRVE divides the objects into four status hierarchies including pointing, in focus, activated, extinct, and step by step it processes multimodal reference resolution according to current status hierarchy. Also it defines a match function to compute the match probability of referring expression and potential referent, and describes the semantic signification and temporal constraints. Finally, sense shape is used to deal with the pointing ambiguity, which helps the user to interact precisely in immersive virtual environment.