How consistent are human judgments of whether an open resource is educational material?

Authors:
Michael C. Harris;James A. Thom;Falk Scholer
Affiliations:
RMIT University, Melbourne, Victoria, Australia;RMIT University, Melbourne, Victoria, Australia;RMIT University, Melbourne, Victoria, Australia
Venue:
ADC '10 Proceedings of the Twenty-First Australasian Conference on Database Technologies - Volume 104
Year:
2010

Citing 8
Cited 0

Information filtering and information retrieval: two sides of the same coin?

Communications of the ACM - Special issue on information filtering
Variations in relevance judgments and the evaluation of retrieval performance

Information Processing and Management: an International Journal
Variations in relevance assessments and the measurement of retrieval effectiveness

Journal of the American Society for Information Science - Special issue: evaluation of information retrieval systems
Variations in relevance judgments and the measurement of retrieval effectiveness

Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval
Reusable learning objects: a survey of LOM-based repositories

Proceedings of the tenth ACM international conference on Multimedia
CLEF Methodology and Metrics

CLEF '01 Revised Papers from the Second Workshop of the Cross-Language Evaluation Forum on Evaluation of Cross-Language Information Retrieval Systems
Toward the Next Generation of Recommender Systems: A Survey of the State-of-the-Art and Possible Extensions

IEEE Transactions on Knowledge and Data Engineering
Relevance: A review of the literature and a framework for thinking on the notion in information science. Part II: nature and manifestations of relevance

Journal of the American Society for Information Science and Technology

Quantified Score

Hi-index	0.00

Visualization

Abstract

Systems that filter web search results to return open educational resources need evaluation. The Cranfield method, which is widely used in information retrieval evaluation, can be used as the basis of a model for evaluating such systems. The Cranfield method requires a collection of resources with associated judgments. In this paper, we describe an experiment to collect judgments of whether open resources are educational material. We show experimentally that judges can agree on what resources are educational material, even in the absence of an educational context, and demonstrate that displaying the query used to retrieve the resource makes a judge less likely to rate a resource as educational.