Adaptive linear information retrieval models
SIGIR '87 Proceedings of the 10th annual international ACM SIGIR conference on Research and development in information retrieval
Measuring relevance judgements
Information Processing and Management: an International Journal
Linear structure in information retrieval
SIGIR '88 Proceedings of the 11th annual international ACM SIGIR conference on Research and development in information retrieval
Optimum polynomial retrieval functions based on the probability ranking principle
ACM Transactions on Information Systems (TOIS)
Determining the effectiveness of retrieval algorithms
Information Processing and Management: an International Journal
Measuring retrieval effectiveness based on user preference of documents
Journal of the American Society for Information Science
Towards the identification of the optimal number of relevance categories
Journal of the American Society for Information Science
From highly relevant to not relevant: examining different regions of relevance
Information Processing and Management: an International Journal
Evaluating evaluation measure stability
SIGIR '00 Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval
IR evaluation methods for retrieving highly relevant documents
SIGIR '00 Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval
Rank aggregation methods for the Web
Proceedings of the 10th international conference on World Wide Web
Journal of the American Society for Information Science and Technology
Information Retrieval
Cumulated gain-based evaluation of IR techniques
ACM Transactions on Information Systems (TOIS)
Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval
Top-k learning to rank: labeling, ranking and evaluation
SIGIR '12 Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval
VANET IR-CAS: utilizing IR techniques in developing context aware system for VANET
Proceedings of the second ACM international symposium on Design and analysis of intelligent vehicular networks and applications
On 3D object retrieval benchmarking
3D Research
| Hi-index | 0.00 |
One of the challenges of modern information retrieval is to rank the most relevant documents at the top of the large system output. This calls for choosing the proper methods to evaluate the system performance. The traditional performance measures, such as precision and recall, are based on binary relevance judgment and are not appropriate for multi-grade relevance. The main objective of this paper is to propose a framework for system evaluation based on user preference of documents. It is shown that the notion of user preference is general and flexible for formally defining and interpreting multi-grade relevance. We review 12 evaluation methods and compare their similarities and differences. We find that the normalized distance performance measure is a good choice in terms of the sensitivity to document rank order and gives higher credits to systems for their ability to retrieve highly relevant documents.