Tasks, topics and relevance judging for the TREC Genomics Track: five years of experience evaluating biomedical text information retrieval systems

  • Authors:

  • Phoebe M. Roberts;Aaron M. Cohen;William R. Hersh

  • Affiliations:

  • Pfizer Research Technology Center, Cambridge, USA 02139;Department of Medical Informatics and Clinical Epidemiology, School of Medicine, Oregon Health & Science University, Portland, USA 97239-3098;Department of Medical Informatics and Clinical Epidemiology, School of Medicine, Oregon Health & Science University, Portland, USA 97239-3098

  • Venue:
  • Information Retrieval
  • Year:
  • 2009

Quantified Score

Hi-index 0.00

Visualization

Abstract

With the help of a team of expert biologist judges, the TREC Genomics track has generated four large sets of "gold standard" test collections, comprised of over a hundred unique topics, two kinds of ad hoc retrieval tasks, and their corresponding relevance judgments. Over the years of the track, increasingly complex tasks necessitated the creation of judging tools and training guidelines to accommodate teams of part-time short-term workers from a variety of specialized biological scientific backgrounds, and to address consistency and reproducibility of the assessment process. Important lessons were learned about factors that influenced the utility of the test collections including topic design, annotations provided by judges, methods used for identifying and training judges, and providing a central moderator "meta-judge".