Analyses for elucidating current question answering technology

  • Authors:

  • Marc Light;Gideon S. Mann;Ellen Riloff;Eric Breck

  • Affiliations:

  • The MITRE Corporation, 202 Burlington Road, Bedford, MA 01730, USA e-mail: light@mitre.org;Department of Computer Science, Johns Hopkins University, Baltimore, MD 21218, USA e-mail: gsm@cs.jhu.edu;School of Computing, University of Utah, Salt Lake City, UT 84112, USA e-mail: riloff@cs.utah.edu;Department of Computer Science, Cornell University, 4161 Upson Hall, Ithaca, NY 14853, USA e-mail: ebreck@cs.cornell.edu

  • Venue:
  • Natural Language Engineering
  • Year:
  • 2001

Quantified Score

Hi-index 0.00

Visualization

Abstract

In this paper, we take a detailed look at the performance of components of an idealized question answering system on two different tasks: the TREC Question Answering task and a set of reading comprehension exams. We carry out three types of analysis: inherent properties of the data, feature analysis, and performance bounds. Based on these analyses we explain some of the performance results of the current generation of Q/A systems and make predictions on future work. In particular, we present four findings: (1) Q/A system performance is correlated with answer repetition; (2) relative overlap scores are more effective than absolute overlap scores; (3) equivalence classes on scoring functions can be used to quantify performance bounds; and (4) perfect answer typing still leaves a great deal of ambiguity for a Q/A system because sentences often contain several items of the same type.