Integrating word acquisition and referential grounding towards physical world interaction

  • Authors:

  • Rui Fang;Changsong Liu;Joyce Yue Chai

  • Affiliations:

  • Michigan State University, East Lansing, MI, USA;Michigan State University, East Lansing, MI, USA;Michigan State University, East Lansing, MI, USA

  • Venue:
  • Proceedings of the 14th ACM international conference on Multimodal interaction
  • Year:
  • 2012

Quantified Score

Hi-index 0.00

Visualization

Abstract

In language-based interaction between a human and an artificial agent (e.g., robot) in a physical world, because the human and the agent have different knowledge and capabilities in perceiving the shared environment, referential grounding is very difficult. To facilitate such interaction, it is important for the agent to continuously learn and acquire knowledge about the environment through interactions with humans and incorporate the learned knowledge in grounding references from human utterances. To address this issue, this paper presents a graph-based approach for referential grounding and examines how referential grounding and word acquisition influence each other in physical world interaction. Our empirical results have shown that for most words, automated word acquisition through interaction improves referential grounding performance. However, this is not the case for words describing object types, where human supervision is important. Nevertheless, better referential grounding enables more accurate acquisition of word meanings, which in turn further improves grounding performance for references in subsequent utterances.