Building a reusable test collection for question answering

  • Authors:

  • Jimmy Lin;Boris Katz

  • Affiliations:

  • College of Information Studies, Institute for Advanced Computer Studies, University of Maryland, College Park, MD 20742;Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, MA 02139

  • Venue:
  • Journal of the American Society for Information Science and Technology - Research Articles
  • Year:
  • 2006

Quantified Score

Hi-index 0.00

Visualization

Abstract

In contrast to traditional information retrieval systems, which return ranked lists of documents that users must manually browse through, a question answering system attempts to directly answer natural language questions posed by the user. Although such systems possess language-processing capabilities, they still rely on traditional document retrieval techniques to generate an initial candidate set of documents. In this article, the authors argue that document retrieval for question answering represents a task different from retrieving documents in response to more general retrospective information needs. Thus, to guide future system development, specialized question answering test collections must be constructed. They show that the current evaluation resources have major shortcomings; to remedy the situation, they have manually created a small, reusable question answering test collection for research purposes. In this article they describe their methodology for building this test collection and discuss issues they encountered regarding the notion of “answer correctness.” © 2006 Wiley Periodicals, Inc.