Why Are Multimodal Systems so Difficult to Build? - About the Difference between Deictic Gestures and Direct Manipulation

Authors:
Michael Streit
Affiliations:
-
Venue:
CMC '98 Revised Papers from the Second International Conference on Cooperative Multimodal Communication
Year:
1998

Citing 7
Cited 2

User and discourse models for multimodal communication

Intelligent user interfaces
Automatic referent resolution of deictic and anaphoric expressions

Computational Linguistics
Integration and synchronization of input modes during multimodal human-computer interaction

Proceedings of the ACM SIGCHI Conference on Human factors in computing systems
Ten myths of multimodal interaction

Communications of the ACM
Intelligent User Interfaces

Intelligent User Interfaces
Active and passive gestures: problems with the resolution of deictic and elliptic expressions in a multimodal system

ReferringPhenomena '97 Referring Phenomena in a Multimedia Context and their Computational Treatment
A syndetic approach to referring phenomena in multimodal interaction

ReferringPhenomena '97 Referring Phenomena in a Multimedia Context and their Computational Treatment

Spatial knowledge representation for human-robot interaction

Spatial cognition III
Human-centered visualization environments

Human-centered visualization environments

Quantified Score

Hi-index	0.00

Visualization

Abstract

By integrating conversationally used natural language with graphical interfaces, gestural interaction looses the simplicity of direct manipulation. Input in one channel may change the meaning of input in another channel rather drastically. This introduces the wait problem, i.e. the problem of when and how long the system has to wait for input in other channels, before it triggers an action.A dditionally, pointing devices and feedback may cause obtrusive effects on the natural synchronisation of deictic gestures and deictic expressions. Certain uses of gestures are observed that are distinct from natural communication and direct manipulation as well, e.g. focusing gestures.T he discussion of these problems is based on our experience with the multimodal prototypes Mofa and Talky that we implemented in different variations, each showing a slightly different style of interaction. The notion of passive and active gesture forms is introduced as well as the notion of active and passive objects. Incremental natural language interpretation and the provision of incremental and preliminary feedback turn out to be important challenges for the upcoming technology of multimodal interfaces.