References to graphical objects in interactive multimodal queries

  • Authors:

  • Daqing He;Graeme Ritchie;John Lee

  • Affiliations:

  • School of Information Sciences, University of Pittsburgh, 135 North Bellefield Avenue, Pittsburgh, PA 15260, USA;Department of Computing Science, University of Aberdeen, Aberdeen AB24 3UE, Scotland, UK;School of Informatics, University of Edinburgh, 2 Buccleuch Place, Edinburgh EH8 9LW, Scotland, UK

  • Venue:
  • Knowledge-Based Systems
  • Year:
  • 2008

Quantified Score

Hi-index 0.00

Visualization

Abstract

Traditionally, interactive natural language systems assume a semantic model in which the entities referred to are in some abstract representation of a real or imagined world. In a system where graphical objects such as diagrams may be on the screen during the language interaction, there is possibility that the user may want to allude to these visual entities. Since graphical objects have their own properties (colour, shape, position on the screen, etc.) but may also represent items in a knowledge base which have other associated properties (price, geographical location, technical specifications, etc.), some systematic way is needed to enable such objects to be referred to in terms of either their screen properties or their associated attributes from the domain under discussion. In this paper, we present a formalisation for these arrangements, and show how our logical definitions can be used to generate constraints suitable for reference resolution within a natural language interpreter.