Using multimodal interaction to navigate in arbitrary virtual VRML worlds

Authors:
Frank Althoff;Gregor McGlaun;Björn Schuller;Peter Morguet;Manfred Lang
Affiliations:
Technical University of Munich, Munich, Germany;Technical University of Munich, Munich, Germany;Technical University of Munich, Munich, Germany;Technical University of Munich, Munich, Germany;Technical University of Munich, Munich, Germany
Venue:
Proceedings of the 2001 workshop on Perceptive user interfaces
Year:
2001

Citing 5
Cited 6

Multimodal interfaces

Artificial Intelligence Review - Special issue on integration of natural language and vision processing: recent advances
Ten myths of multimodal interaction

Communications of the ACM
Usability Engineering

Usability Engineering
Virtual Notepad: Handwriting in Immersive VR

VRAIS '98 Proceedings of the Virtual Reality Annual International Symposium
Comparison of approaches to continuous hand gesture recognition for a visual dialog system

ICASSP '99 Proceedings of the Acoustics, Speech, and Signal Processing, 1999. on 1999 IEEE International Conference - Volume 06

A generic approach for interfacing VRML browsers to various input devices and creating customizable 3D applications

Proceedings of the seventh international conference on 3D Web technology
A user interface framework for multimodal VR interactions

ICMI '05 Proceedings of the 7th international conference on Multimodal interfaces
Semantic 3D object manipulation using object ontology in multimodal interaction framework

Proceedings of the 2005 international conference on Augmented tele-existence
Empowering followers in virtual teams: Guiding principles from theory and practice

Computers in Human Behavior
A multidimensional dynamic time warping algorithm for efficient multimodal fusion of asynchronous data streams

Neurocomputing
Verbal control of mathematical tools for simulation and virtual environments

Proceedings of the 2010 Summer Computer Simulation Conference

Quantified Score

Hi-index	0.02

Visualization

Abstract

In this paper we present a multimodal interface for navigating in arbitrary virtual VRML worlds. Conventional haptic devices like keyboard, mouse, joystick and touchscreen can freely be combined with special Virtual-Reality hardware like spacemouse, data glove and position tracker. As a key feature, the system additionally provides intuitive input by command and natural speech utterances as well as dynamic head and hand gestures. The commuication of the interface components is based on the abstract formalism of a context-free grammar, allowing the representation of device-independent information. Taking into account the current system context, user interactions are combined in a semantic unification process and mapped on a model of the viewer's functionality vocabulary. To integrate the continuous multimodal information stream we use a straight-forward rule-based approach and a new technique based on evolutionary algorithms. Our navigation interface has extensively been evaluated in usability studies, obtaining excellent results.