An end-to-end evaluation of two situated dialog systems

  • Authors:
  • Lina M. Rojas-Barahona;Alejandra Lorenzo;Claire Gardent

  • Affiliations:
  • Inria, LORIA, UMR, France;Université de Lorraine, LORIA, UMR, France;CNRS, LORIA, UMR, France

  • Venue:
  • SIGDIAL '12 Proceedings of the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue
  • Year:
  • 2012

Quantified Score

Hi-index 0.00

Visualization

Abstract

We present and evaluate two state-of-the art dialogue systems developed to support dialog with French speaking virtual characters in the context of a serious game: one hybrid statistical/symbolic and one purely statistical. We conducted a quantitative evaluation where we compare the accuracy of the interpreter and of the dialog manager used by each system; a user based evaluation based on 22 subjects using both the statistical and the hybrid system; and a corpus based evaluation where we examine such criteria as dialog coherence, dialog success, interpretation and generation errors in the corpus of Human-System interactions collected during the user-based evaluation. We show that although the statistical approach is slightly more robust, the hybrid strategy seems to be better at guiding the player through the game.