Synergistic use of direct manipulation and natural language
CHI '89 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Discourse theory and interface design: the case of pointing with the mouse
International Journal of Man-Machine Studies
Designing the user interface (2nd ed.): strategies for effective human-computer interaction
Designing the user interface (2nd ed.): strategies for effective human-computer interaction
Intelligent multimedia interfaces
Intelligent multimedia interfaces
The measurement of computer literacy: a comparison of self-appraisal and objective tests
International Journal of Human-Computer Studies
ISSD-93 Selected papers presented at the international symposium on Spoken dialogue
A third modality of natural language?
Artificial Intelligence Review - Special issue on integration of natural language and vision processing: intelligent multimedia
Communications of the ACM
Multimodal interfaces for dynamic interactive maps
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Integration and synchronization of input modes during multimodal human-computer interaction
Proceedings of the ACM SIGCHI Conference on Human factors in computing systems
User Centered System Design; New Perspectives on Human-Computer Interaction
User Centered System Design; New Perspectives on Human-Computer Interaction
Empirical studies of discourse representations for natural language interfaces
EACL '89 Proceedings of the fourth conference on European chapter of the Association for Computational Linguistics
Natural and simulated pointing
EACL '87 Proceedings of the third conference on European chapter of the Association for Computational Linguistics
Generating multimodal output: conditions, advantages and problems
COLING '88 Proceedings of the 12th conference on Computational linguistics - Volume 2
Referring in multimodal systems: the importance of user expertise and system features
ReferringPhenomena '97 Referring Phenomena in a Multimedia Context and their Computational Treatment
Ecological Interfaces: Extending the Pointing Paradigm by Visual Context
CONTEXT '99 Proceedings of the Second International and Interdisciplinary Conference on Modeling and Using Context
The Ecological Approach to Multimodal System Design
GW '99 Proceedings of the International Gesture Workshop on Gesture-Based Communication in Human-Computer Interaction
Explorative studies on multimodal interaction in a PDA- and desktop-based scenario
ICMI '08 Proceedings of the 10th international conference on Multimodal interfaces
| Hi-index | 0.01 |
This paper examines user behavior during multimodal human-computer interaction (HCI). It discusses how pointing, natural language, and graphical layout should be integrated to enhance the usability of multimodal systems. Two experiments were run to study simulated systems capable of understanding written natural language and mouse-supported pointing gestures. Results allowed to: (a) develop a taxonomy of communication acts aimed at identifying targets; (b) determine the conditions under which specific referent identification strategies are likely to be produced; (c) suggest guidelines for designing effective multimodal interfaces; (d) show that performance is strongly influenced by interface graphical layout and by user expertise. Our study confirms the value of simulation as a tool for building HCI models and supports the basic idea that linguistic, visual, and motor cues can be integrated to favor effective multimodal communication.