A virtual protocol model for computer-human interaction
International Journal of Man-Machine Studies
Integrating simultaneous input from speech, gaze, and hand gestures
Intelligent multimedia interfaces
A design space for multimodal systems: concurrent processing and data fusion
INTERCHI '93 Proceedings of the INTERCHI '93 conference on Human factors in computing systems
Artificial Intelligence Review - Special issue on integration of natural language and vision processing: recent advances
Communications of the ACM
QuickSet: multimodal interaction for distributed applications
MULTIMEDIA '97 Proceedings of the fifth ACM international conference on Multimedia
Ten myths of multimodal interaction
Communications of the ACM
Perceptual user interfaces: multimodal interfaces that process what comes naturally
Communications of the ACM
Multimodal human discourse: gesture and speech
ACM Transactions on Computer-Human Interaction (TOCHI)
“Put-that-there”: Voice and gesture at the graphics interface
SIGGRAPH '80 Proceedings of the 7th annual conference on Computer graphics and interactive techniques
Advances in Robust Multimodal Interface Design
IEEE Computer Graphics and Applications
Proceedings of the 5th international conference on Multimodal interfaces
Guidelines for multimodal user interface design
Communications of the ACM - Multimodal interfaces that flex, adapt, and persist
A probabilistic approach to reference resolution in multimodal user interfaces
Proceedings of the 9th international conference on Intelligent user interfaces
Unification-based multimodal integration
ACL '98 Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics and Eighth Conference of the European Chapter of the Association for Computational Linguistics
Unification-based multimodal parsing
COLING '98 Proceedings of the 17th international conference on Computational linguistics - Volume 1
When do we interact multimodally?: cognitive load and multimodal communication patterns
Proceedings of the 6th international conference on Multimodal interfaces
Individual differences in multimodal integration patterns: what are they and why do they exist?
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Finite-state multimodal integration and understanding
Natural Language Engineering
Natural language with integrated deictic and graphic gestures
HLT '89 Proceedings of the workshop on Speech and Natural Language
Multimodal fusion: a new hybrid strategy for dialogue systems
Proceedings of the 8th international conference on Multimodal interfaces
Multimodal human-computer interaction: A survey
Computer Vision and Image Understanding
Multimodal Interaction with Mobile Devices: Fusing a Broad Spectrum of Modality Combinations
Multimodal Interaction with Mobile Devices: Fusing a Broad Spectrum of Modality Combinations
Usability framework for the design and evaluation of multimodal interaction
BCS-HCI '08 Proceedings of the 22nd British HCI Group Annual Conference on People and Computers: Culture, Creativity, Interaction - Volume 2
Multimodal integration: a biological view
IJCAI'01 Proceedings of the 17th international joint conference on Artificial intelligence - Volume 2
Fusion engines for multimodal input: a survey
Proceedings of the 2009 international conference on Multimodal interfaces
A fusion framework for multimodal interactive applications
Proceedings of the 2009 international conference on Multimodal interfaces
Facilitating multiparty dialog with gaze, gesture, and speech
International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction
Multimodal integration-a statistical view
IEEE Transactions on Multimedia
Multimodal framework for mobile interaction
Proceedings of the International Working Conference on Advanced Visual Interfaces
Statistical multimodal integration for audio-visual speech processing
IEEE Transactions on Neural Networks
Integrating the physical environment into mobile remote collaboration
MobileHCI '12 Proceedings of the 14th international conference on Human-computer interaction with mobile devices and services
Multimodal human behavior analysis: learning correlation and interaction across modalities
Proceedings of the 14th ACM international conference on Multimodal interaction
Multimodal behavior and interaction as indicators of cognitive load
ACM Transactions on Interactive Intelligent Systems (TiiS) - Special issue on highlights of the decade in interactive intelligent systems
Hi-index | 0.10 |
People naturally interact with the world multimodally, through both parallel and sequential use of multiple perceptual modalities. Multimodal human-computer interaction has sought for decades to endow computers with similar capabilities, in order to provide more natural, powerful, and compelling interactive experiences. With the rapid advance in non-desktop computing generated by powerful mobile devices and affordable sensors in recent years, multimodal research that leverages speech, touch, vision, and gesture is on the rise. This paper provides a brief and personal review of some of the key aspects and issues in multimodal interaction, touching on the history, opportunities, and challenges of the area, especially in the area of multimodal integration. We review the question of early vs. late integration and find inspiration in recent evidence in biological sensory integration. Finally, we list challenges that lie ahead for research in multimodal human-computer interaction.