A hand gesture interface device
CHI '87 Proceedings of the SIGCHI/GI Conference on Human Factors in Computing Systems and Graphics Interface
Processing Description containing Words and Gestures -A System Architecture-
GWAI-86 und 2. Österreichische Artificial-Intelligence-Tagung
Ein erster Blick auf POPEL - Wie wird was gesagt?
GWAI '87 Proceedings of the 11th German Workshop on Artificial Intelligence
Generating natural language under pragmatic constraints
Generating natural language under pragmatic constraints
Natural and simulated pointing
EACL '87 Proceedings of the third conference on European chapter of the Association for Computational Linguistics
Commenting on action: continuous linguistic feedback generation
IUI '93 Proceedings of the 1st international conference on Intelligent user interfaces
Visual display, pointing, and natural language: the power of multimodal interaction
AVI '98 Proceedings of the working conference on Advanced visual interfaces
| Hi-index | 0.00 |
In natural communication situations, multimodel referent specification is frequent and efficient. The linguistic component are deictic expressions, e.g. 'this and 'here'. Extralinguistic devices in dialogs are different body movements, mainly pointing gestures. Their functional equivalent in texts are means like arrows and indices.This paper has two intentions. First, it discusses the advantages of multimodal reference in interhuman communication which motivate the integration of extralinguistic "pointing" devices into NL dialog systems. The generation of multimodal output poses specific problems, which have no counterpart in the analysis of multimodal input. The second part presents the strategy for generating multimodal output which has been developed within the framework of the XTRA system (a NL access system to expert systems). XTRA allows the combination of verbal descriptions and pointing gestures in order to specify elements of the given visual context, i.e. a form displayed on the screen. The component POPEL generates referential expressions which may be accompanied by a pointing gesture. The appearance of these gestures depends on several factors, e.g. the type of referent (whether it is a region or an entry of the form) and its complexity.