QuickSet: multimodal interaction for distributed applications

Authors:
Philip R. Cohen;Michael Johnston;David McGee;Sharon Oviatt;Jay Pittman;Ira Smith;Liang Chen;Josh Clow
Affiliations:
Center for Human Computer Communication, Oregon Graduate Institute of Science and Technology, P.O. Box 91000 Portland, OR;Center for Human Computer Communication, Oregon Graduate Institute of Science and Technology, P.O. Box 91000 Portland, OR;Center for Human Computer Communication, Oregon Graduate Institute of Science and Technology, P.O. Box 91000 Portland, OR;Center for Human Computer Communication, Oregon Graduate Institute of Science and Technology, P.O. Box 91000 Portland, OR;Center for Human Computer Communication, Oregon Graduate Institute of Science and Technology, P.O. Box 91000 Portland, OR;Center for Human Computer Communication, Oregon Graduate Institute of Science and Technology, P.O. Box 91000 Portland, OR;Center for Human Computer Communication, Oregon Graduate Institute of Science and Technology, P.O. Box 91000 Portland, OR;Center for Human Computer Communication, Oregon Graduate Institute of Science and Technology, P.O. Box 91000 Portland, OR
Venue:
MULTIMEDIA '97 Proceedings of the fifth ACM international conference on Multimedia
Year:
1997

Citing 18
Cited 154

Synergistic use of direct manipulation and natural language

CHI '89 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Recognizing handwritten text

CHI '91 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Intelligent multi-media interface technology

Intelligent user interfaces
The logic of typed feature structures

The logic of typed feature structures
The role of natural language in a multimodal interface

UIST '92 Proceedings of the 5th annual ACM symposium on User interface software and technology
NPSNET: constructing a 3D virtual world

I3D '92 Proceedings of the 1992 symposium on Interactive 3D graphics
Integrating simultaneous input from speech, gaze, and hand gestures

Intelligent multimedia interfaces
Surround-screen projection-based virtual reality: the design and implementation of the CAVE

SIGGRAPH '93 Proceedings of the 20th annual conference on Computer graphics and interactive techniques
Placing text labels on maps and diagrams

Graphics gems IV
Virtual reality on a WIM: interactive worlds in miniature

CHI '95 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Multimodal interfaces for dynamic interactive maps

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Multimodal interfaces

Artificial Intelligence Review - Special issue on integration of natural language and vision processing: recent advances
Integration and synchronization of input modes during multimodal human-computer interaction

Proceedings of the ACM SIGCHI Conference on Human factors in computing systems
“Put-that-there”: Voice and gesture at the graphics interface

SIGGRAPH '80 Proceedings of the 7th annual conference on Computer graphics and interactive techniques
CommandTalk: a spoken-language interface for battlefield simulations

ANLC '97 Proceedings of the fifth conference on Applied natural language processing
Unification-based multimodal integration

ACL '98 Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics and Eighth Conference of the European Chapter of the Association for Computational Linguistics
Building an application framework for speech and pen input integration in multimodal learning interfaces

ICASSP '96 Proceedings of the Acoustics, Speech, and Signal Processing, 1996. on Conference Proceedings., 1996 IEEE International Conference - Volume 06
Multimodal interactive maps: designing for human performance

Human-Computer Interaction

Gesture VR: vision-based 3D hand interace for spatial interaction

MULTIMEDIA '98 Proceedings of the sixth ACM international conference on Multimedia
A robust selection system using real-time multi-modal user-agent interactions

IUI '99 Proceedings of the 4th international conference on Intelligent user interfaces
Design principles for intelligent environments

AAAI '98/IAAI '98 Proceedings of the fifteenth national/tenth conference on Artificial intelligence/Innovative applications of artificial intelligence
Mutual disambiguation of recognition errors in a multimodel architecture

Proceedings of the SIGCHI conference on Human Factors in Computing Systems
Ten myths of multimodal interaction

Communications of the ACM
Perceptual user interfaces: multimodal interfaces that process what comes naturally

Communications of the ACM
Towards a fault-tolerant multi-agent system architecture

AGENTS '00 Proceedings of the fourth international conference on Autonomous agents
Taming recognition errors with a multimodal interface

Communications of the ACM
Multimodal system processing in mobile environments

UIST '00 Proceedings of the 13th annual ACM symposium on User interface software and technology
Something from nothing: augmenting a paper-based work practice via multimodal interaction

DARE '00 Proceedings of DARE 2000 on Designing augmented reality environments
Towards a computational model of sketching

Proceedings of the 6th international conference on Intelligent user interfaces
Creating tangible interfaces by augmenting physical objects with multimodal language

Proceedings of the 6th international conference on Intelligent user interfaces
Hands-free multi-scale navigation in virtual environments

I3D '01 Proceedings of the 2001 symposium on Interactive 3D graphics
Monitoring deployed agent teams

Proceedings of the fifth international conference on Autonomous agents
Join and capture: a model for nomadic interaction

Proceedings of the 14th annual ACM symposium on User interface software and technology
Sketching for knowledge capture: a progress report

Proceedings of the 7th international conference on Intelligent user interfaces
Comparing paper and tangible, multimodal tools

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Speech and gesture multimodal control of a whole Earth 3D visualization environment

VISSYM '02 Proceedings of the symposium on Data Visualisation 2002
Channeled multicast for group communications

Proceedings of the first international joint conference on Autonomous agents and multiagent systems: part 3
Designing robust multimodal systems for universal access

WUAUC'01 Proceedings of the 2001 EC/NSF workshop on Universal accessibility of ubiquitous computing: providing for the elderly
Towards heterogeneous agent teams

Mutli-agents systems and applications
A voice and ink XML multimodal architecture for mobile e-commerce systems

WMC '02 Proceedings of the 2nd international workshop on Mobile commerce
Distributed mediation of ambiguous context in aware environments

Proceedings of the 15th annual ACM symposium on User interface software and technology
Designing a human-centered, multimodal GIS interface to support emergency management

Proceedings of the 10th ACM international symposium on Advances in geographic information systems
Evaluation of a multimodal interface for 3D terrain visualization

Proceedings of the conference on Visualization '02
Sketching for military courses of action diagrams

Proceedings of the 8th international conference on Intelligent user interfaces
Speech/Gesture Interface to a Visual-Computing Environment

IEEE Computer Graphics and Applications
Immersive VR for Scientific Visualization: A Progress Report

IEEE Computer Graphics and Applications
Partial Information in Multimodal Dialogue

ICMI '00 Proceedings of the Third International Conference on Advances in Multimodal Interfaces
Towards Heterogeneous Agent Teams

EASSS '01 Selected Tutorial Papers from the 9th ECCAI Advanced Course ACAI 2001 and Agent Link's 3rd European Agent Systems Summer School on Multi-Agent Systems and Applications
A Multimodal Interface Framework for Using Hand Gestures and Speech in Virtual Environment Applications

GW '99 Proceedings of the International Gesture Workshop on Gesture-Based Communication in Human-Computer Interaction
Real-Time Gesture Recognition by Means of Hybrid Recognizers

GW '01 Revised Papers from the International Gesture Workshop on Gesture and Sign Languages in Human-Computer Interaction
A reference model for multimodal input interpretation

CHI '03 Extended Abstracts on Human Factors in Computing Systems
Multimodal interfaces

The human-computer interaction handbook
AI on the battlefield: an experimental exploration

Eighteenth national conference on Artificial intelligence
Advances in the robust processing of multimodal speech and pen systems

Multimodal interface for human-machine communication
Multimodal Interaction During Multiparty Dialogues: Initial Results

ICMI '02 Proceedings of the 4th IEEE International Conference on Multimodal Interfaces
A Map-Based System Using Speech and 3D Gestures for Pervasive Computing

ICMI '02 Proceedings of the 4th IEEE International Conference on Multimodal Interfaces
Context-Based Multimodal Input Understanding in Conversational Systems

ICMI '02 Proceedings of the 4th IEEE International Conference on Multimodal Interfaces
Flexi-Modal and Multi-Machine User Interfaces

ICMI '02 Proceedings of the 4th IEEE International Conference on Multimodal Interfaces
Incorporating AI into Military Decision Making: An Experiment

IEEE Intelligent Systems
Advances in Robust Multimodal Interface Design

IEEE Computer Graphics and Applications
Mutual disambiguation of 3D multimodal interaction in augmented and virtual reality

Proceedings of the 5th international conference on Multimodal interfaces
A framework for rapid development of multimodal interfaces

Proceedings of the 5th international conference on Multimodal interfaces
Capturing user tests in a multimodal, multidevice informal prototyping tool

Proceedings of the 5th international conference on Multimodal interfaces
Modeling multimodal integration patterns and performance in seniors: toward adaptive processing of individual differences

Proceedings of the 5th international conference on Multimodal interfaces
Tangible multimodal interfaces for safety-critical applications

Communications of the ACM - Multimodal interfaces that flex, adapt, and persist
A probabilistic approach to reference resolution in multimodal user interfaces

Proceedings of the 9th international conference on Intelligent user interfaces
The connected user interface: realizing a personal situated navigation service

Proceedings of the 9th international conference on Intelligent user interfaces
Visually prototyping perceptual user interfaces through multimodal storyboarding

Proceedings of the 2001 workshop on Perceptive user interfaces
The efficiency of multimodal interaction for a map-based task

ANLC '00 Proceedings of the sixth conference on Applied natural language processing
Unification-based multimodal parsing

COLING '98 Proceedings of the 17th international conference on Computational linguistics - Volume 1
Visual and linguistic information in gesture classification

Proceedings of the 6th international conference on Multimodal interfaces
A model-based approach for real-time embedded multimodal systems in military aircrafts

Proceedings of the 6th international conference on Multimodal interfaces
Context based multimodal fusion

Proceedings of the 6th international conference on Multimodal interfaces
A multimodal learning interface for sketch, speak and point creation of a schedule chart

Proceedings of the 6th international conference on Multimodal interfaces
The CommandTalk spoken dialogue system

ACL '99 Proceedings of the 37th annual meeting of the Association for Computational Linguistics on Computational Linguistics
Linguistic theories in efficient multimodal reference resolution: an empirical investigation

Proceedings of the 10th international conference on Intelligent user interfaces
Multimodal interaction for pedestrians: an evaluation study

Proceedings of the 10th international conference on Intelligent user interfaces
Animating an interactive conversational character for an educational game system

Proceedings of the 10th international conference on Intelligent user interfaces
Qualitative spatial reasoning about sketch maps

AI Magazine
ICARE: a component-based approach for multimodal interaction

UbiMob '04 Proceedings of the 1st French-speaking conference on Mobility and ubiquity computing
Designing mediation for context-aware applications

ACM Transactions on Computer-Human Interaction (TOCHI)
From text to images through meanings

Proceedings of the 2004 ACM SIGCHI International Conference on Advances in computer entertainment technology
Semantics-based representation for multimodal interpretation in conversational systems

COLING '02 Proceedings of the 19th international conference on Computational linguistics - Volume 1
A framework for device capability on demand and virtual device user experience

IBM Journal of Research and Development
Distributed pointing for multimodal collaboration over sketched diagrams

ICMI '05 Proceedings of the 7th international conference on Multimodal interfaces
A study of manual gesture-based selection for the PEMMI multimodal transport management interface

ICMI '05 Proceedings of the 7th international conference on Multimodal interfaces
Modality preferences in mobile and instrumented environments

Proceedings of the 11th international conference on Intelligent user interfaces
Combining semantic and temporal constraints for multimodal integration in conversation systems

HLT-NAACL-DIALOGUE '03 Proceedings of the HLT-NAACL 2003 workshop on Research directions in dialogue processing - Volume 7
Enabling interaction with single user applications through speech and gestures on a multi-user tabletop

Proceedings of the working conference on Advanced visual interfaces
Fluid inking: augmenting the medium of free-form inking with gestures

GI '06 Proceedings of Graphics Interface 2006
Utilisation de SMV pour la vérification de propriétés d'IHM multimodales

IHM 2004 Proceedings of the 16th conference on Association Francophone d'Interaction Homme-Machine
Multimodal user interface facilitating critical data entry for traffic incident management

MMUI '05 Proceedings of the 2005 NICTA-HCSNet Multimodal User Interaction Workshop - Volume 57
Human-centered collaborative interaction

Proceedings of the 1st ACM international workshop on Human-centered multimedia
Collaborative multimodal photo annotation over digital paper

Proceedings of the 8th international conference on Multimodal interfaces
GSI demo: multiuser gesture/speech interaction over digital tables by wrapping single user applications

Proceedings of the 8th international conference on Multimodal interfaces
Using redundant speech and handwriting for learning new vocabulary and understanding abbreviations

Proceedings of the 8th international conference on Multimodal interfaces
Visual and linguistic information in gesture classification

ACM SIGGRAPH 2006 Courses
Optimization in multimodal interpretation

ACL '04 Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics
A salience driven approach to robust input interpretation in multimodal conversational systems

HLT '05 Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing
Multimodal multiplayer tabletop gaming

Computers in Entertainment (CIE) - Interactive TV
Visual and linguistic information in gesture classification

ACM SIGGRAPH 2007 courses
Interaction framework for home environment using speech and vision

Image and Vision Computing
Multimodal human-computer interaction: A survey

Computer Vision and Image Understanding
A tactile language for intuitive human-robot communication

Proceedings of the 9th international conference on Multimodal interfaces
VoicePen: augmenting pen input with simultaneous non-linguisitic vocalization

Proceedings of the 9th international conference on Multimodal interfaces
Integrated speech and gaze control for realistic desktop environments

Proceedings of the 2008 symposium on Eye tracking research & applications
Strengths and weaknesses of software architectures for the rapid creation of tangible and multimodal interfaces

Proceedings of the 2nd international conference on Tangible and embedded interaction
Efficiency of speech recognition for using interface design environments by novel designers

AIC'07 Proceedings of the 7th Conference on 7th WSEAS International Conference on Applied Informatics and Communications - Volume 7
Multimodal interaction systems: information and time features

International Journal of Web and Grid Services
Personalised maps in multimodal mobile GIS

International Journal of Web Engineering and Technology
Implicit interaction for pro-active assistance in a context-adaptive warehouse application

Mobility '07 Proceedings of the 4th international conference on mobile technology, applications, and systems and the 1st international symposium on Computer human interaction in mobile technology
Beyond attention: the role of deictic gesture in intention recognition in multimodal conversational interfaces

Proceedings of the 13th international conference on Intelligent user interfaces
Speech and sketching: an empirical study of multimodal interaction

SBIM '07 Proceedings of the 4th Eurographics workshop on Sketch-based interfaces and modeling
HCI Beyond the GUI: Design for Haptic, Speech, Olfactory, and Other Nontraditional Interfaces

HCI Beyond the GUI: Design for Haptic, Speech, Olfactory, and Other Nontraditional Interfaces
Exploring true multi-user multimodal interaction over a digital table

Proceedings of the 7th ACM conference on Designing interactive systems
Sketch recognition in interspersed drawings using time-based graphical models

Computers and Graphics
A Distributed Wearable System Based on Multimodal Fusion

ICESS '07 Proceedings of the 3rd international conference on Embedded Software and Systems
Towards a minimalist multimodal dialogue framework using recursive MVC pattern

ICMI '08 Proceedings of the 10th international conference on Multimodal interfaces
A Wizard of Oz study for an AR multimodal interface

ICMI '08 Proceedings of the 10th international conference on Multimodal interfaces
Steps in Identifying Interaction Design Patterns for Multimodal Systems

HCSE-TAMODIA '08 Proceedings of the 2nd Conference on Human-Centered Software Engineering and 7th International Workshop on Task Models and Diagrams
Designing the user interface for multimodal speech and pen-based gesture applications: state-of-the-art systems and future research directions

Human-Computer Interaction
"Move the couch where?": developing an augmented reality multimodal interface

ISMAR '06 Proceedings of the 5th IEEE and ACM International Symposium on Mixed and Augmented Reality
How effective is it to design by voice?

BCS-HCI '07 Proceedings of the 21st British HCI Group Annual Conference on People and Computers: HCI...but not as we know it - Volume 2
Multimodal Interfaces: A Survey of Principles, Models and Frameworks

Human Machine Interaction
MEMODULES as Tangible Shortcuts to Multimedia Information

Human Machine Interaction
An open source workbench for prototyping multimodal interactions based on off-the-shelf heterogeneous components

Proceedings of the 1st ACM SIGCHI symposium on Engineering interactive computing systems
Iconic and multi-stroke gesture recognition

Pattern Recognition
Reliable Evaluation of Multimodal Dialogue Systems

Proceedings of the 13th International Conference on Human-Computer Interaction. Part II: Novel Interaction Methods and Techniques
Performance evaluation and error analysis for multimodal reference resolution in a conversation system

HLT-NAACL-Short '04 Proceedings of HLT-NAACL 2004: Short Papers
Analogical learning of visual/conceptual relationships in sketches

AAAI'05 Proceedings of the 20th national conference on Artificial intelligence - Volume 1
Cognitive principles in robust multimodal interpretation

Journal of Artificial Intelligence Research
Gesture salience as a hidden variable for coreference resolution and keyframe extraction

Journal of Artificial Intelligence Research
Monitoring teams by overhearing: a multi-agent plan-recognition approach

Journal of Artificial Intelligence Research
Multimodal interaction: a new focal area for AI

IJCAI'01 Proceedings of the 17th international joint conference on Artificial intelligence - Volume 2
Between linguistic attention and gaze fixations inmultimodal conversational interfaces

Proceedings of the 2009 international conference on Multimodal interfaces
Fusion engines for multimodal input: a survey

Proceedings of the 2009 international conference on Multimodal interfaces
Benchmarking fusion engines of multimodal interactive systems

Proceedings of the 2009 international conference on Multimodal interfaces
Formal description techniques to support the design, construction and evaluation of fusion engines for sure (safe, usable, reliable and evolvable) multimodal interfaces

Proceedings of the 2009 international conference on Multimodal interfaces
AI on the battlefield: an experimental exploration

IAAI'02 Proceedings of the 14th conference on Innovative applications of artificial intelligence - Volume 1
Combining speech and sketch to interpret unconstrained descriptions of mechanical devices

IJCAI'09 Proceedings of the 21st international jont conference on Artifical intelligence
Dynamic reconfiguration of multimodal architectures

CompSysTech '09 Proceedings of the International Conference on Computer Systems and Technologies and Workshop for PhD Students in Computing
Adding speech recognition support to UML tools

Journal of Visual Languages and Computing
Manual deskterity: an exploration of simultaneous pen + touch direct input

CHI '10 Extended Abstracts on Human Factors in Computing Systems
The recognition and comprehension of hand gestures: a review and research agenda

ZiF'06 Proceedings of the Embodied communication in humans and machines, 2nd ZiF research group international conference on Modeling communication with robots and virtual humans
New ideas on the artificial intelligence support in military applications

AIKED'10 Proceedings of the 9th WSEAS international conference on Artificial intelligence, knowledge engineering and data bases
Gesture interaction in cooperation scenarios

CRIWG'09 Proceedings of the 15th international conference on Groupware: design, implementation, and use
COMM notation for specifying collaborative and multimodal interactive systems

Proceedings of the 2nd ACM SIGCHI symposium on Engineering interactive computing systems
Pen + touch = new tools

UIST '10 Proceedings of the 23nd annual ACM symposium on User interface software and technology
Toward natural interaction in the real world: real-time gesture recognition

International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction
Component-based high fidelity interactive prototyping of post-WIMP interactions

International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction
Vision-based hand-gesture applications

Communications of the ACM
Design and evaluation of a multimodal interface for in-car communication systems

SAICSIT '10 Proceedings of the 2010 Annual Research Conference of the South African Institute of Computer Scientists and Information Technologists
Human-centered visualization environments

Human-centered visualization environments
Multimodal, touchless interaction in spatial augmented reality environments

ICDHM'11 Proceedings of the Third international conference on Digital human modeling
A multimodal reference resolution approach in virtual environment

VSMM'06 Proceedings of the 12th international conference on Interactive Technologies and Sociotechnical Systems
Research on user-centered design and recognition pen gestures

CGI'06 Proceedings of the 24th international conference on Advances in Computer Graphics
Transcription table: text support during meetings

INTERACT'05 Proceedings of the 2005 IFIP TC13 international conference on Human-Computer Interaction
An evaluation of an augmented reality multimodal interface using speech and paddle gestures

ICAT'06 Proceedings of the 16th international conference on Advances in Artificial Reality and Tele-Existence
Just do what i tell you: the limited impact of instructions on multimodal integration patterns

UM'05 Proceedings of the 10th international conference on User Modeling
Integrating intra and extra gestures into a mobile and multimodal shopping assistant

PERVASIVE'05 Proceedings of the Third international conference on Pervasive Computing
IntellWheels MMI: a flexible interface for an intelligent wheelchair

RoboCup 2009
Test of the ICARE platform fusion mechanism

DSVIS'05 Proceedings of the 12th international conference on Interactive Systems: design, specification, and verification
A multimodal fusion framework for children's storytelling systems

Edutainment'06 Proceedings of the First international conference on Technologies for E-Learning and Digital Entertainment
Multimodal framework for mobile interaction

Proceedings of the International Working Conference on Advanced Visual Interfaces
A distributed staged architecture for multimodal applications

ECSA'07 Proceedings of the First European conference on Software Architecture
Finger-based multitouch interface for performing 3D CAD operations

International Journal of Human-Computer Studies
Using the transferable belief model for multimodal input fusion in companion systems

MPRSS'12 Proceedings of the First international conference on Multimodal Pattern Recognition of Social Signals in Human-Computer-Interaction
Formal description of multi-touch interactions

Proceedings of the 5th ACM SIGCHI symposium on Engineering interactive computing systems
OrMiS: a tabletop interface for simulation-based training

Proceedings of the 2013 ACM international conference on Interactive tabletops and surfaces
Empirical evaluation of multimodal input interactions

HCI International'13 Proceedings of the 15th international conference on Human Interface and the Management of Information: information and interaction design - Volume Part I
Demonstration of sketch-thru-plan: a multimodal interface for command and control

Proceedings of the 15th ACM on International conference on multimodal interaction
Review Article: Multimodal interaction: A review

Pattern Recognition Letters

Quantified Score

Hi-index	0.06

QuickSet: multimodal interaction for distributed applications

Quantified Score

Visualization

Abstract