Taking advantage of the situation: non-linguistic context for natural language interfaces to interactive virtual environments

  • Authors:
  • Michael Fleischman;Eduard Hovy

  • Affiliations:
  • Massachusetts Institute of Technology;University of Southern California

  • Venue:
  • Proceedings of the 11th international conference on Intelligent user interfaces
  • Year:
  • 2006

Quantified Score

Hi-index 0.00

Visualization

Abstract

We introduce a framework for learning situated Natural Language Interfaces (NLIs) to interactive virtual environments. The framework exploits the non-linguistic context, or situation, explicitly modeled in such interactive applications. This situation model is integrated with a model of word meaning in a principled manner using a noisy channel approach to language understanding. Preliminary experimentation in an independently designed interactive application, i.e. the Mission Rehearsal Exercise (MRE), shows that this situated NLI outperforms a state of the art NLI on both whole frame accuracy and F-Score metrics. Further, use of the situation model in the situated NLI is shown to increase robustness to the noise introduced by the use of automatic speech recognition.