Variations in relevance judgments and the measurement of retrieval effectiveness
Information Processing and Management: an International Journal
Cumulated gain-based evaluation of IR techniques
ACM Transactions on Information Systems (TOIS)
Information retrieval system evaluation: effort, sensitivity, and reliability
Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval
Reliability tests for the XCG and inex-2002 metrics
INEX'04 Proceedings of the Third international conference on Initiative for the Evaluation of XML Retrieval
IIiX Proceedings of the 1st international conference on Information interaction in context
Investigating the exhaustivity dimension in content-oriented XML element retrieval evaluation
CIKM '06 Proceedings of the 15th ACM international conference on Information and knowledge management
Evaluating XML retrieval effectiveness at INEX
ACM SIGIR Forum
Sound and complete relevance assessment for XML retrieval
ACM Transactions on Information Systems (TOIS)
Focused Access to XML Documents
XML Retrieval by Improving Structural Relevance Measures Obtained from Summary Models
Focused Access to XML Documents
The Simplest XML Retrieval Baseline That Could Possibly Work
Focused Access to XML Documents
On the evaluation of snippet selection for WebCLEF
CLEF'08 Proceedings of the 9th Cross-language evaluation forum conference on Evaluating systems for multilingual and multimodal information access
RMIT university at INEX 2005: ad hoc track
INEX'05 Proceedings of the 4th international conference on Initiative for the Evaluation of XML Retrieval
Combining image and structured text retrieval
INEX'05 Proceedings of the 4th international conference on Initiative for the Evaluation of XML Retrieval
| Hi-index | 0.00 |
This paper describes our proposal for an evaluation metric for XML retrieval that is solely based on the highlighted text. We support our decision of ignoring the exhaustivity dimension by undertaking a critical investigation of the two INEX 2005 relevance dimensions. We present a fine grained empirical analysis of the level of assessor agreement of the five topics double-judged at INEX 2005, and show that the agreement is higher for specificity than for exhaustivity. We use the proposed metric to evaluate the INEX 2005 runs for each retrieval strategy of the CO and CAS retrieval tasks. A correlation analysis of the rank orderings obtained by the new metric and two XCG metrics shows that the orderings are strongly correlated, which demonstrates the usefulness of the proposed metric for evaluation of XML retrieval performance.