An evaluation of an augmented reality multimodal interface using speech and paddle gestures

Authors:
Sylvia Irawati;Scott Green;Mark Billinghurst;Andreas Duenser;Heedong Ko
Affiliations:
Imaging Media Research Center, Korea Institute of Science and Technology;Human Interface Technology Laboratory New Zealand, University of Canterbury;Human Interface Technology Laboratory New Zealand, University of Canterbury;Human Interface Technology Laboratory New Zealand, University of Canterbury;Imaging Media Research Center, Korea Institute of Science and Technology
Venue:
ICAT'06 Proceedings of the 16th international conference on Advances in Artificial Reality and Tele-Existence
Year:
2006

Citing 18
Cited 6

Synergistic use of direct manipulation and natural language

CHI '89 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Speech and gestures for graphic image manipulation

CHI '89 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Augmenting a Window System with Speech Input

Computer
Talk and Draw: Bundling Speech and Graphics

Computer
The role of natural language in a multimodal interface

UIST '92 Proceedings of the 5th annual ACM symposium on User interface software and technology
Speech-driven graphics: a user interface

Journal of Microcomputer Applications - Special issue on computer applications for handicapped persons
QuickSet: multimodal interaction for distributed applications

MULTIMEDIA '97 Proceedings of the fifth ACM international conference on Multimedia
Creating tangible interfaces by augmenting physical objects with multimodal language

Proceedings of the 6th international conference on Intelligent user interfaces
Multimodal Interaction for 2D and 3D Environments

IEEE Computer Graphics and Applications
“Put-that-there”: Voice and gesture at the graphics interface

SIGGRAPH '80 Proceedings of the 7th annual conference on Computer graphics and interactive techniques
SenseShapes: Using Statistical Geometry for Object Selection in a Multimodal Augmented Reality System

ISMAR '03 Proceedings of the 2nd IEEE/ACM International Symposium on Mixed and Augmented Reality
Mutual disambiguation of 3D multimodal interaction in augmented and virtual reality

Proceedings of the 5th international conference on Multimodal interfaces
The magic wand

SCCG '03 Proceedings of the 19th spring conference on Computer graphics
FingARtips: gesture based direct manipulation in Augmented Reality

Proceedings of the 2nd international conference on Computer graphics and interactive techniques in Australasia and South East Asia
Multimodal interaction in an augmented reality scenario

Proceedings of the 6th international conference on Multimodal interfaces
Implementation and evaluation of a constraint-based multimodal fusion system for speech and 3D pointing gestures

Proceedings of the 6th international conference on Multimodal interfaces
Rapid prototyping for spoken dialogue systems

COLING '02 Proceedings of the 19th international conference on Computational linguistics - Volume 1
Augmented reality agents in the development pipeline of computer entertainment

ICEC'05 Proceedings of the 4th international conference on Entertainment Computing

A Wizard of Oz study for an AR multimodal interface

ICMI '08 Proceedings of the 10th international conference on Multimodal interfaces
Identification et prise en compte de propriétés ergonomiques pour la modélisation et la conception de SIM

Proceedings of the 20th International Conference of the Association Francophone d'Interaction Homme-Machine
Usability recommendations in the design of mixed interactive systems

Proceedings of the 1st ACM SIGCHI symposium on Engineering interactive computing systems
Advanced Interaction Techniques for Augmented Reality Applications

VMR '09 Proceedings of the 3rd International Conference on Virtual and Mixed Reality: Held as Part of HCI International 2009
Just say it: an evaluation of speech interfaces for augmented reality design applications

AICS'09 Proceedings of the 20th Irish conference on Artificial intelligence and cognitive science
Hands and speech in space: multimodal interaction with augmented reality interfaces

Proceedings of the 15th ACM on International conference on multimodal interaction

Quantified Score

Hi-index	0.02

Visualization

Abstract

This paper discusses an evaluation of an augmented reality (AR) multimodal interface that uses combined speech and paddle gestures for interaction with virtual objects in the real world. We briefly describe our AR multimodal interface architecture and multimodal fusion strategies that are based on the combination of time-based and domain semantics. Then, we present the results from a user study comparing using multimodal input to using gesture input alone. The results show that a combination of speech and paddle gestures improves the efficiency of user interaction. Finally, we describe some design recommendations for developing other multimodal AR interfaces.