Effect of written instructions on assessor agreement

Authors:
William Webber;Bryan Toth;Marjorie Desamito
Affiliations:
University of Maryland, College Park, MD, USA;Eleanor Roosevelt High School, Greenbelt, MD, USA;Eleanor Roosevelt High School, Greenbelt, MD, USA
Venue:
SIGIR '12 Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval
Year:
2012

Citing 2
Cited 3

Variations in relevance judgments and the measurement of retrieval effectiveness

Information Processing and Management: an International Journal
Document categorization in legal electronic discovery: computer classification vs. manual review

Journal of the American Society for Information Science and Technology

Alternative assessor disagreement and retrieval depth

Proceedings of the 21st ACM international conference on Information and knowledge management
The effect of threshold priming and need for cognition on relevance calibration and assessment

Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval
Document features predicting assessor disagreement

Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval

Quantified Score

Hi-index	0.00

Visualization

Abstract

Assessors frequently disagree on the topical relevance of documents. How much of this disagreement is due to ambiguity in assessment instructions? We have two assessors assess TREC Legal Track documents for relevance, some to a general topic description, others to detailed assessment guidelines. We find that detailed guidelines lead to no significant increase in agreement amongst assessors or between assessors and the official qrels.