Construction of a test collection for the focussed retrieval of structured documents

  • Authors:

  • Gabriella Kazai;Mounia Lalmas;Jane Reid

  • Affiliations:

  • Department of Computer Science, Queen Mary, University of London, London;Department of Computer Science, Queen Mary, University of London, London;Department of Computer Science, Queen Mary, University of London, London

  • Venue:
  • ECIR'03 Proceedings of the 25th European conference on IR research
  • Year:
  • 2003

Quantified Score

Hi-index 0.00

Visualization

Abstract

In this paper, we examine the methodological issues involved in constructing test collections of structured documents and obtaining best entry points for the evaluation of the focussed retrieval of document components. We describe a pilot test of the proposed test collection construction methodology performed on a document collection of Shakespeare plays. In our analysis, we examine the effect of query complexity and type on overall query difficulty, the use of multiple relevance judges for each query, the problem of obtaining exhaustive relevance assessments from participants, and the method of eliciting relevance assessments and best entry points. Our findings indicate that the methodology is indeed feasible in this small-scale context, and merits further investigation.