Looking under the hood: tools for diagnosing your question answering engine

Authors:
Eric Breck;Marc Light;Gideon S. Mann;Ellen Riloff;Brianne Brown;Pranav Anand;Mats Rooth;Michael Thelen
Affiliations:
The MITRE Corporation, Bedford, MA;The MITRE Corporation, Bedford, MA;Johns Hopkins University, Baltimore;University of Utah, Salt Lake City, UT;Bryn Mawr College, Bryn Mawr, PA;Harvard University, Cambridge, MA;Cornell University, Ithaca, NY;University of Utah, Salt Lake City, UT
Venue:
ODQA '01 Proceedings of the workshop on Open-domain question answering - Volume 12
Year:
2001

Citing 5
Cited 4

Deep Read: a reading comprehension system

ACL '99 Proceedings of the 37th annual meeting of the Association for Computational Linguistics on Computational Linguistics
Reading comprehension programs in a statistical-language-processing class

ANLP/NAACL-ReadingComp '00 Proceedings of the 2000 ANLP/NAACL Workshop on Reading comprehension tests as evaluation for computer-based language understanding sytems - Volume 6
A rule-based question answering system for reading comprehension tests

ANLP/NAACL-ReadingComp '00 Proceedings of the 2000 ANLP/NAACL Workshop on Reading comprehension tests as evaluation for computer-based language understanding sytems - Volume 6
A question answering system developed as a project in a natural language processing course

ANLP/NAACL-ReadingComp '00 Proceedings of the 2000 ANLP/NAACL Workshop on Reading comprehension tests as evaluation for computer-based language understanding sytems - Volume 6
A machine learning approach to answering questions for reading comprehension tests

EMNLP '00 Proceedings of the 2000 Joint SIGDAT conference on Empirical methods in natural language processing and very large corpora: held in conjunction with the 38th Annual Meeting of the Association for Computational Linguistics - Volume 13

Performance issues and error analysis in an open-domain question answering system

ACM Transactions on Information Systems (TOIS)
Question answering from the web using knowledge annotation and knowledge mining techniques

CIKM '03 Proceedings of the twelfth international conference on Information and knowledge management
Mining natural language answers from the web

Web Intelligence and Agent Systems
A statistical method for short answer extraction

ODQA '01 Proceedings of the workshop on Open-domain question answering - Volume 12

Quantified Score

Hi-index	0.00

Visualization

Abstract

In this paper we analyze two question answering tasks: the TREC-8 question answering task and a set of reading comprehension exams. First, we show that Q/A systems perform better when there are multiple answer opportunities per question. Next, we analyze common approaches to two subproblems: term overlap for answer sentence identification, and answer typing for short answer extraction. We present general tools for analyzing the strengths and limitations of techniques for these sub-problems. Our results quantify the limitations of both term overlap and answer typing to distinguish between competing answer candidates.