Speech and gesture multimodal control of a whole Earth 3D visualization environment

Authors:
David M. Krum;Olugbenga Omoteso;William Ribarsky;Thad Starner;Larry F. Hodges
Affiliations:
Georgia Institute of Technology, Atlanta, GA;Georgia Institute of Technology, Atlanta, GA;Georgia Institute of Technology, Atlanta, GA;Georgia Institute of Technology, Atlanta, GA;Georgia Institute of Technology, Atlanta, GA
Venue:
VISSYM '02 Proceedings of the symposium on Data Visualisation 2002
Year:
2002

Citing 13
Cited 10

Evaluation of the CyberGlove as a whole-hand input device

ACM Transactions on Computer-Human Interaction (TOCHI)
Moving objects in space: exploiting proprioception in virtual-environment interaction

Proceedings of the 24th annual conference on Computer graphics and interactive techniques
QuickSet: multimodal interaction for distributed applications

MULTIMEDIA '97 Proceedings of the fifth ACM international conference on Multimedia
Mutual disambiguation of recognition errors in a multimodel architecture

Proceedings of the SIGCHI conference on Human Factors in Computing Systems
Multimodal Interaction for 2D and 3D Environments

IEEE Computer Graphics and Applications
“Put-that-there”: Voice and gesture at the graphics interface

SIGGRAPH '80 Proceedings of the 7th annual conference on Computer graphics and interactive techniques
The Software Architecture of a Real-Time Battlefield Visualization Virtual Environment

VR '99 Proceedings of the IEEE Virtual Reality
Third-Person Navigation of Whole-Planet Terrain in a Head-tracked Stereoscopic Environment

VR '99 Proceedings of the IEEE Virtual Reality
Real-Time Input of 3D Pose and Gestures of a User's Hand and Its Applications for HCI

VR '01 Proceedings of the Virtual Reality 2001 Conference (VR'01)
Speech/Gesture Interface to a Visual Computing Environment for Molecular Biologists

ICPR '96 Proceedings of the International Conference on Pattern Recognition (ICPR '96) Volume III-Volume 7276 - Volume 7276
The Gesture Pendant: A Self-illuminating, Wearable, Infrared Computer Vision System for Home Automation Control and Medical Monitoring

ISWC '00 Proceedings of the 4th IEEE International Symposium on Wearable Computers
Interaction techniques for common tasks in immersive virtual environments: design, evaluation, and application

Interaction techniques for common tasks in immersive virtual environments: design, evaluation, and application
Designing the user interface for multimodal speech and pen-based gesture applications: state-of-the-art systems and future research directions

Human-Computer Interaction

Evaluation of a multimodal interface for 3D terrain visualization

Proceedings of the conference on Visualization '02
Mutual disambiguation of 3D multimodal interaction in augmented and virtual reality

Proceedings of the 5th international conference on Multimodal interfaces
Multimodal Interaction with a Wearable Augmented Reality System

IEEE Computer Graphics and Applications
A 3D interactive kiosk system

AFRIGRAPH '07 Proceedings of the 5th international conference on Computer graphics, virtual reality, visualisation and interaction in Africa
User evaluation of god-like interaction techniques

AUIC '08 Proceedings of the ninth conference on Australasian user interface - Volume 76
Implementation of god-like interaction techniques for supporting collaboration between outdoor AR and indoor tabletop users

ISMAR '06 Proceedings of the 5th IEEE and ACM International Symposium on Mixed and Augmented Reality
Adding speech recognition support to UML tools

Journal of Visual Languages and Computing
GeeAir: a universal multimodal remote control device for home appliances

Personal and Ubiquitous Computing
Human-centered visualization environments

Human-centered visualization environments
Hands and speech in space: multimodal interaction with augmented reality interfaces

Proceedings of the 15th ACM on International conference on multimodal interaction

Quantified Score

Hi-index	0.00

Visualization

Abstract

A growing body of research shows several advantages to multimodal interfaces including increased expressiveness, flexibility, and user freedom. This paper investigates the design of such an interface that integrates speech and hand gestures. The interface has the additional property of operating relative to the user and can be used while the user is in motion or standing at a distance from the computer display. The paper then describes an implementation of the multimodal interface for a whole Earth 3D visualization which presents navigation interface challenges due to the large magnitude of scale and extended spaces that are available. The characteristics of the multimodal interface are examined, such as speed, recognizability of gestures, ease and accuracy of use, and learnability under likely conditions of use. This implementation shows that such a multimodal interface can be effective in a real environment and sets some parameters for the design and use of such interfaces.