Variations in relevance judgments and the measurement of retrieval effectiveness
Information Processing and Management: an International Journal
The effect of assessor error on IR system evaluation
Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval
Assessor error in stratified evaluation
CIKM '10 Proceedings of the 19th ACM international conference on Information and knowledge management
Measuring assessor accuracy: a comparison of nist assessors and user study participants
Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval
| Hi-index | 0.00 |
When human assessors judge documents for their relevance to a search topic, it is possible for errors in judging to occur. As part of the analysis of the data collected from a 48 participant user study, we have discovered that when the participants made relevance judgments, the average participant spent more time to make errorful judgments than to make correct judgments. Thus, in relevance assessing scenarios similar to our user study, it may be possible to use the time taken to judge a document as an indicator of assessor error. Such an indicator could be used to identify documents that are candidates for adjudication or reassessment.