A critical investigation of recall and precision as measures of retrieval system performance
ACM Transactions on Information Systems (TOIS)
Using statistical testing in the evaluation of retrieval experiments
SIGIR '93 Proceedings of the 16th annual international ACM SIGIR conference on Research and development in information retrieval
Statistical inference in retrieval effectiveness evaluation
Information Processing and Management: an International Journal
The effect of topic set size on retrieval experiment error
SIGIR '02 Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval
Cumulated gain-based evaluation of IR techniques
ACM Transactions on Information Systems (TOIS)
The overlap problem in content-oriented XML retrieval evaluation
Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval
Providing consistent and exhaustive relevance assessments for XML retrieval evaluation
Proceedings of the thirteenth ACM international conference on Information and knowledge management
Information retrieval system evaluation: effort, sensitivity, and reliability
Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval
HiXEval: highlighting XML retrieval evaluation
INEX'05 Proceedings of the 4th international conference on Initiative for the Evaluation of XML Retrieval
INEX'04 Proceedings of the Third international conference on Initiative for the Evaluation of XML Retrieval
Evaluating XML retrieval effectiveness at INEX
ACM SIGIR Forum
Advances in XML retrieval: the INEX initiative
Proceedings of the 2006 international workshop on Research issues in digital libraries
Sound and complete relevance assessment for XML retrieval
ACM Transactions on Information Systems (TOIS)
Building a framework for the probability ranking principle by a family of expected weighted rank
ACM Transactions on Information Systems (TOIS)
INEX 2002-2006: understanding XML retrieval evaluation
DELOS'07 Proceedings of the 1st international conference on Digital libraries: research and development
Contextualization models for XML retrieval
Information Processing and Management: an International Journal
| Hi-index | 0.00 |
INEX, the evaluation initiative for content-oriented XML retrieval, has since its establishment defined the relevance of an element according to two graded dimensions, exhaustivity and specificity. The former measures how exhaustively an XML element discusses the topic of request, whereas specificity measures how focused the element is on the topic of request. The reason for having two dimensions was to provide a more stable measure of relevance than if assessors were asked to rate the relevance of an element on a single scale. However, obtaining relevance assessments is a costly task. as each document must be assessed for relevance by a human assessor. In XML retrieval this problem is exacerbated as the elements of the document must also be assessed with respect to the exhaustivity and specificity dimensions. A continuous discussion in INEX has been whether such a sophisticated definition of relevance, and in particular the exhaustivity dimension, was needed. This paper attempts to answer this question through extensive statistical tests to compare the conclusions about system performance that could be made under different assessment scenarios.