Towards a general theory of action and time
Artificial Intelligence
The use of hand-drawn gestures for text editing
International Journal of Man-Machine Studies
Speech and gestures for graphic image manipulation
CHI '89 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
A tutorial on hidden Markov models and selected applications in speech recognition
Readings in speech recognition
COMET: generating coordinated multimedia explanations
CHI '91 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Combining visual and acoustic speech signals with a neural network improves intelligibility
Advances in neural information processing systems 2
Intelligent multi-media interface technology
Intelligent user interfaces
Graphics and natural language as components of automatic explanation
Intelligent user interfaces
The logic of typed feature structures
The logic of typed feature structures
Interactive simulation in a multi-person virtual world
CHI '92 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Multimodal interaction in speech systems
Multimedia interface design
Plan-based integration of natural language and graphics generation
Artificial Intelligence - Special volume on natural language processing
Watch what I do: programming by demonstration
Watch what I do: programming by demonstration
Using the Baby-Babble-Blanket for infants with motor problems: an empirical study
Assets '94 Proceedings of the first annual ACM conference on Assistive technologies
Voice communication between humans and machines
Voice communication between humans and machines
The role of voice in human-machine communication
Voice communication between humans and machines
Cooperating heterogeneous systems: a blackboard-based meta approach
Cooperating heterogeneous systems: a blackboard-based meta approach
Speech recognition in noisy environments: a survey
Speech Communication
ISSD-93 Selected papers presented at the international symposium on Spoken dialogue
Interactive sketching for the early stages of user interface design
CHI '95 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Artificial Intelligence Review - Special issue on integration of natural language and vision processing: recent advances
Sketching storyboards to illustrate interface behaviors
Conference Companion on Human Factors in Computing Systems
Integration and synchronization of input modes during multimodal human-computer interaction
Proceedings of the ACM SIGCHI Conference on Human factors in computing systems
MedSpeak: report creation with continuous speech recognition
Proceedings of the ACM SIGCHI Conference on Human factors in computing systems
The CSLU toolkit: rapid prototyping of spoken language systems
Proceedings of the 10th annual ACM symposium on User interface software and technology
QuickSet: multimodal interaction for distributed applications
MULTIMEDIA '97 Proceedings of the fifth ACM international conference on Multimedia
Proceedings of the third international ACM conference on Assistive technologies
The 3rd ACM SIGCAPH Conference on Assistive Technologies
3rd IEEE workshop on interactive voice technology for telecommunications applications
Speech Communication - Special issue on interactive voice technology for telecommunication applications (IVITA '96)
Survey of the state of the art in human language technology
Survey of the state of the art in human language technology
Readings in agents
Synergistic use of direct manipulation and natural language
Readings in intelligent user interfaces
User-centered modeling for spoken language and multimodal interfaces
Readings in intelligent user interfaces
Design principles for intelligent environments
AAAI '98/IAAI '98 Proceedings of the fifteenth national/tenth conference on Artificial intelligence/Innovative applications of artificial intelligence
Speech Communication - Special issue on auditory-visual speech processing
Adaptive fusion of acoustic and visual sources for automatic speech recognition
Speech Communication - Special issue on auditory-visual speech processing
A pragmatic principle for agent communication
Proceedings of the third annual conference on Autonomous Agents
Principles of mixed-initiative user interfaces
Proceedings of the SIGCHI conference on Human Factors in Computing Systems
Manual and gaze input cascaded (MAGIC) pointing
Proceedings of the SIGCHI conference on Human Factors in Computing Systems
Patterns of entry and correction in large vocabulary continuous speech recognition systems
Proceedings of the SIGCHI conference on Human Factors in Computing Systems
Mutual disambiguation of recognition errors in a multimodel architecture
Proceedings of the SIGCHI conference on Human Factors in Computing Systems
Ten myths of multimodal interaction
Communications of the ACM
Perceptual user interfaces (introduction)
Communications of the ACM
Perceptual user interfaces: multimodal interfaces that process what comes naturally
Communications of the ACM
Towards a fault-tolerant multi-agent system architecture
AGENTS '00 Proceedings of the fourth international conference on Autonomous agents
Taming recognition errors with a multimodal interface
Communications of the ACM
Suede: a Wizard of Oz prototyping tool for speech user interfaces
UIST '00 Proceedings of the 13th annual ACM symposium on User interface software and technology
Multimodal system processing in mobile environments
UIST '00 Proceedings of the 13th annual ACM symposium on User interface software and technology
Something from nothing: augmenting a paper-based work practice via multimodal interaction
DARE '00 Proceedings of DARE 2000 on Designing augmented reality environments
An iterative design methodology for user-friendly natural language office information applications
ACM Transactions on Information Systems (TOIS)
Creating tangible interfaces by augmenting physical objects with multimodal language
Proceedings of the 6th international conference on Intelligent user interfaces
Plan Recognition in Natural Language Dialogue
Plan Recognition in Natural Language Dialogue
Planning English Sentences
Multimodal Interaction for 2D and 3D Environments
IEEE Computer Graphics and Applications
Designing the user interface for pen and speech multimedia applications
CHI '99 Extended Abstracts on Human Factors in Computing Systems
A Unified Framework for Constructing Multimodal Experiments and Applications
CMC '98 Revised Papers from the Second International Conference on Cooperative Multimodal Communication
ECAI '96 Proceedings of the Workshop on Intelligent Agents III, Agent Theories, Architectures, and Languages
“Put-that-there”: Voice and gesture at the graphics interface
SIGGRAPH '80 Proceedings of the 7th annual conference on Computer graphics and interactive techniques
Integration of audio/visual information for use in human-computer intelligent interaction
ICIP '97 Proceedings of the 1997 International Conference on Image Processing (ICIP '97) 3-Volume Set-Volume 1 - Volume 1
ISWC '97 Proceedings of the 1st IEEE International Symposium on Wearable Computers
A framework and toolkit for the construction of multimodal learning interfaces
A framework and toolkit for the construction of multimodal learning interfaces
Unification-based multimodal integration
ACL '98 Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics and Eighth Conference of the European Chapter of the Association for Computational Linguistics
Confirmation in multimodal systems
COLING '98 Proceedings of the 17th international conference on Computational linguistics - Volume 2
Unification-based multimodal parsing
COLING '98 Proceedings of the 17th international conference on Computational linguistics - Volume 1
Adaptive bimodal sensor fusion for automatic speechreading
ICASSP '96 Proceedings of the Acoustics, Speech, and Signal Processing, 1996. on Conference Proceedings., 1996 IEEE International Conference - Volume 02
ICASSP '96 Proceedings of the Acoustics, Speech, and Signal Processing, 1996. on Conference Proceedings., 1996 IEEE International Conference - Volume 06
Multimodal interactive maps: designing for human performance
Human-Computer Interaction
Multimodal integration-a statistical view
IEEE Transactions on Multimedia
From members to teams to committee-a robust approach to gestural and multimodal recognition
IEEE Transactions on Neural Networks
Perceptual user interfaces: multimodal interfaces that process what comes naturally
Communications of the ACM
Taming recognition errors with a multimodal interface
Communications of the ACM
EGVE '02 Proceedings of the workshop on Virtual environments 2002
Speech and gesture multimodal control of a whole Earth 3D visualization environment
VISSYM '02 Proceedings of the symposium on Data Visualisation 2002
Designing robust multimodal systems for universal access
WUAUC'01 Proceedings of the 2001 EC/NSF workshop on Universal accessibility of ubiquitous computing: providing for the elderly
A voice and ink XML multimodal architecture for mobile e-commerce systems
WMC '02 Proceedings of the 2nd international workshop on Mobile commerce
Using marking menus to develop command sets for computer vision based hand gesture interfaces
Proceedings of the second Nordic conference on Human-computer interaction
Teaching mathematical explanation through audiographic technology
Computers & Education
Comparison of Various Interface Modalities for a Locomotion Assistance Device
ICCHP '02 Proceedings of the 8th International Conference on Computers Helping People with Special Needs
IHM '02 Proceedings of the 14th French-speaking conference on Human-computer interaction (Conférence Francophone sur l'Interaction Homme-Machine)
Advances in the robust processing of multimodal speech and pen systems
Multimodal interface for human-machine communication
Designing Transition Networks for Multimodal VR-Interactions Using a Markup Language
ICMI '02 Proceedings of the 4th IEEE International Conference on Multimodal Interfaces
Embarking on Multimodal Interface Design
ICMI '02 Proceedings of the 4th IEEE International Conference on Multimodal Interfaces
Mobile Multi-Modal Data Services for GPRS Phones and Beyond
ICMI '02 Proceedings of the 4th IEEE International Conference on Multimodal Interfaces
Advances in Robust Multimodal Interface Design
IEEE Computer Graphics and Applications
Capturing user tests in a multimodal, multidevice informal prototyping tool
Proceedings of the 5th international conference on Multimodal interfaces
Where is "it"? Event Synchronization in Gaze-Speech Input Systems
Proceedings of the 5th international conference on Multimodal interfaces
Augmenting user interfaces with adaptive speech commands
Proceedings of the 5th international conference on Multimodal interfaces
Error recovery in a blended style eye gaze and speech interface
Proceedings of the 5th international conference on Multimodal interfaces
A visually grounded natural language interface for reference to spatial scenes
Proceedings of the 5th international conference on Multimodal interfaces
The role of spoken feedback in experiencing multimodal interfaces as human-like
Proceedings of the 5th international conference on Multimodal interfaces
Proceedings of the 5th international conference on Multimodal interfaces
Speech and sketching for multimodal design
Proceedings of the 9th international conference on Intelligent user interfaces
Naturally conveyed explanations of device behavior
Proceedings of the 2001 workshop on Perceptive user interfaces
ICARE: a component-based approach for the design and development of multimodal interfaces
CHI '04 Extended Abstracts on Human Factors in Computing Systems
When do we interact multimodally?: cognitive load and multimodal communication patterns
Proceedings of the 6th international conference on Multimodal interfaces
ICARE software components for rapidly developing multimodal interfaces
Proceedings of the 6th international conference on Multimodal interfaces
ACM Transactions on Computer-Human Interaction (TOCHI)
An interrogative visualization environment for large-scale engineering simulations
Advances in Engineering Software
An adaptive approach to collecting multimodal input
ACL '03 Proceedings of the 41st Annual Meeting on Association for Computational Linguistics - Volume 2
A conceptual framework for developing adaptive multimodal applications
Proceedings of the 11th international conference on Intelligent user interfaces
Artificial Intelligence - Special volume on connecting language to the world
Combining speech and pen input for effective interaction in mobile geospatial environments
Proceedings of the 2006 ACM symposium on Applied computing
A longitudinal evaluation of hands-free speech-based navigation during dictation
International Journal of Human-Computer Studies
Explicit task representation based on gesture interaction
MMUI '05 Proceedings of the 2005 NICTA-HCSNet Multimodal User Interaction Workshop - Volume 57
Naturally conveyed explanations of device behavior
ACM SIGGRAPH 2006 Courses
Speech and sketching for multimodal design
ACM SIGGRAPH 2006 Courses
Signal Processing - Special section: Multimodal human-computer interfaces
Speech and sketching for multimodal design
ACM SIGGRAPH 2007 courses
Multimodal human-computer interaction: A survey
Computer Vision and Image Understanding
Agora: a GUI approach to multimodal user interfaces
HLT '02 Proceedings of the second international conference on Human Language Technology Research
EURASIP Journal on Applied Signal Processing
VoicePen: augmenting pen input with simultaneous non-linguisitic vocalization
Proceedings of the 9th international conference on Multimodal interfaces
Artificial Intelligence Review
Efficiency of speech recognition for using interface design environments by novel designers
AIC'07 Proceedings of the 7th Conference on 7th WSEAS International Conference on Applied Informatics and Communications - Volume 7
Personalised maps in multimodal mobile GIS
International Journal of Web Engineering and Technology
International Journal of Web Engineering and Technology
Speech and sketching: an empirical study of multimodal interaction
SBIM '07 Proceedings of the 4th Eurographics workshop on Sketch-based interfaces and modeling
HCI Beyond the GUI: Design for Haptic, Speech, Olfactory, and Other Nontraditional Interfaces
HCI Beyond the GUI: Design for Haptic, Speech, Olfactory, and Other Nontraditional Interfaces
Applicability of No-Hands Computer Input Devices for the Certificates for Microsoft Office Software
ICCHP '08 Proceedings of the 11th international conference on Computers Helping People with Special Needs
Towards Specifying Multimodal Collaborative User Interfaces: A Comparison of Collaboration Notations
Interactive Systems. Design, Specification, and Verification
Steps in Identifying Interaction Design Patterns for Multimodal Systems
HCSE-TAMODIA '08 Proceedings of the 2nd Conference on Human-Centered Software Engineering and 7th International Workshop on Task Models and Diagrams
Hands-free, speech-based navigation during dictation: difficulties, consequences, and solutions
Human-Computer Interaction
Parakeet: a continuous speech recognition system for mobile touch-screen devices
Proceedings of the 14th international conference on Intelligent user interfaces
Conception de systèmes collaboratifs multimodaux: analyse comparative de notations
Proceedings of the 20th International Conference of the Association Francophone d'Interaction Homme-Machine
Usability framework for the design and evaluation of multimodal interaction
BCS-HCI '08 Proceedings of the 22nd British HCI Group Annual Conference on People and Computers: Culture, Creativity, Interaction - Volume 2
Multimodal Interfaces: A Survey of Principles, Models and Frameworks
Human Machine Interaction
A dynamic Bayesian approach to computational Laban shape quality analysis
Advances in Human-Computer Interaction
Design of communication in multimodal web interfaces
Proceedings of the 27th ACM international conference on Design of communication
Multimodal interaction: a new focal area for AI
IJCAI'01 Proceedings of the 17th international joint conference on Artificial intelligence - Volume 2
Artificial Intelligence - Special volume on connecting language to the world
Benchmarking fusion engines of multimodal interactive systems
Proceedings of the 2009 international conference on Multimodal interfaces
IEEE Transactions on Audio, Speech, and Language Processing - Special issue on multimodal processing in speech-based interactions
Enhancing input on and above the interactive surface with muscle sensing
Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces
User expectations from dictation on mobile devices
HCI'07 Proceedings of the 12th international conference on Human-computer interaction: interaction platforms and techniques
Effects of multimodal feedback on the performance of older adults with normal and impaired vision
ERCIM'02 Proceedings of the User interfaces for all 7th international conference on Universal access: theoretical perspectives, practice, and experience
Multimodal interfaces for in-vehicle applications
HCI'07 Proceedings of the 12th international conference on Human-computer interaction: intelligent multimodal interaction environments
Designing transparent interaction for ubiquitous computing: theory and application
HCI'07 Proceedings of the 12th international conference on Human-computer interaction: interaction design and usability
What gestures to perform a collaborative storytelling?
ICVS'07 Proceedings of the 4th international conference on Virtual storytelling: using virtual reality technologies for storytelling
Spoken and multimodal communication systems in mobile settings
COST 2102'07 Proceedings of the 2007 COST action 2102 international conference on Verbal and nonverbal communication behaviours
Adaptive user interactive sketching for teaching based on pen gesture
EPCE'07 Proceedings of the 7th international conference on Engineering psychology and cognitive ergonomics
Introducing multimodal paper-digital interfaces for speech-language therapy
Proceedings of the 12th international ACM SIGACCESS conference on Computers and accessibility
'How may I help you'-spoken queries for technical assistance
Proceedings of the 48th Annual Southeast Regional Conference
An advanced multimodal platform for educational social networks
OTM'10 Proceedings of the 2010 international conference on On the move to meaningful internet systems
Multimodal instantiation of assistance services
Proceedings of the 12th International Conference on Information Integration and Web-based Applications & Services
The future of natural user interfaces
CHI '11 Extended Abstracts on Human Factors in Computing Systems
In-car dictation and driver's distraction: a case study
HCII'11 Proceedings of the 14th international conference on Human-computer interaction: towards mobile and intelligent interaction environments - Volume Part III
Write-N-Speak: Authoring Multimodal Digital-Paper Materials for Speech-Language Therapy
ACM Transactions on Accessible Computing (TACCESS)
A comparison between spoken queries and menu-based interfaces for in-car digital music selection
INTERACT'05 Proceedings of the 2005 IFIP TC13 international conference on Human-Computer Interaction
Multimodal human computer interaction: a survey
ICCV'05 Proceedings of the 2005 international conference on Computer Vision in Human-Computer Interaction
A pattern-based methodology for multimodal interaction design
TSD'06 Proceedings of the 9th international conference on Text, Speech and Dialogue
Multimodal architectures: issues and experiences
OTM'06 Proceedings of the 2006 international conference on On the Move to Meaningful Internet Systems: AWeSOMe, CAMS, COMINF, IS, KSinBIT, MIOS-CIAO, MONET - Volume Part I
Delivering personalized context-aware spatial information to mobile devices
W2GIS'05 Proceedings of the 5th international conference on Web and Wireless Geographical Information Systems
Test of the ICARE platform fusion mechanism
DSVIS'05 Proceedings of the 12th international conference on Interactive Systems: design, specification, and verification
TAP & PLAY: an end-user toolkit for authoring interactive pen and paper language activities
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Dictating and editing short texts while driving: distraction and task completion
Proceedings of the 3rd International Conference on Automotive User Interfaces and Interactive Vehicular Applications
Context-based bounding volume morphing in pointing gesture application
HCI'13 Proceedings of the 15th international conference on Human-Computer Interaction: interaction modalities and techniques - Volume Part IV
Review Article: Multimodal interaction: A review
Pattern Recognition Letters
Hi-index | 0.03 |
The growing interest in multimodal interface design is inspired in large part by the goals of supporting more transparent, flexible, efficient, and powerfully expressive means of human-computer interaction than in the past. Multimodal interfaces are expected to support a wider range of diverse applications, be usable by a broader spectrum of the average population, and function more reliably under realistic and challenging usage conditions. In this article, we summarize the emerging architectural approaches for interpreting speech and pen-based gestural input in a robust manner-including early and late fusion approaches, and the new hybrid symbolic-statistical approach. We also describe a diverse collection of state-of-the-art multimodal systems that process users' spoken and gestural input. These applications range from map-based and virtual reality systems for engaging in simulations and training, to field medic systems for mobile use in noisy environments, to web-based transactions and standard text-editing applications that will reshape daily computing and have a significant commercial impact. To realize successful multimodal systems of the future, many key research challenges remain to be addressed. Among these challenges are the development of cognitive theories to guide multimodal system design, and the development of effective natural language processing, dialogue processing, and error-handling techniques. In addition, new multimodal systems will be needed that can function more robustly and adaptively, and with support for collaborative multiperson use. Before this new class of systems can proliferate, toolkits also will be needed to promote software development for both simulated and functioning systems.