Decision theory: an introduction to the mathematics of rationality
Decision theory: an introduction to the mathematics of rationality
Adaptive linear information retrieval models
SIGIR '87 Proceedings of the 10th annual international ACM SIGIR conference on Research and development in information retrieval
Measuring relevance judgements
Information Processing and Management: an International Journal
Optimum polynomial retrieval functions based on the probability ranking principle
ACM Transactions on Information Systems (TOIS)
Extended Boolean query processing in the generalized vector space model
Information Systems
A re-examination of relevance: toward a dynamic, situational definition
Information Processing and Management: an International Journal
Preference structure, inference and set-oriented retrieval
SIGIR '91 Proceedings of the 14th annual international ACM SIGIR conference on Research and development in information retrieval
An analysis of vector space models based on computational geometry
SIGIR '92 Proceedings of the 15th annual international ACM SIGIR conference on Research and development in information retrieval
Probabilistic retrieval based on staged logistic regression
SIGIR '92 Proceedings of the 15th annual international ACM SIGIR conference on Research and development in information retrieval
Probabilistic models in information retrieval
The Computer Journal - Special issue on information retrieval
Variations in relevance judgments and the evaluation of retrieval performance
Information Processing and Management: an International Journal
Term dependence: truncating the Bahadur Lazarsfeld expansion
Information Processing and Management: an International Journal
Measuring retrieval effectiveness based on user preference of documents
Journal of the American Society for Information Science
Human subjectivity and performance limits in document retrieval
Information Processing and Management: an International Journal - Special issue: history of information science
On extending the vector space model for Boolean query processing
Proceedings of the 9th annual international ACM SIGIR conference on Research and development in information retrieval
Dempster-Shafer's theory of evidence applied to structured documents: modelling uncertainty
Proceedings of the 20th annual international ACM SIGIR conference on Research and development in information retrieval
The knowledge in multiple human relevance judgments
ACM Transactions on Information Systems (TOIS)
Users' criteria for relevance evaluation: a cross-situational comparison
Information Processing and Management: an International Journal
A language modeling approach to information retrieval
Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval
“Is this document relevant?…probably”: a survey of probabilistic models in information retrieval
ACM Computing Surveys (CSUR)
On Relevance, Probabilistic Indexing and Information Retrieval
Journal of the ACM (JACM)
A vector space model for automatic indexing
Communications of the ACM
Evaluation by highly relevant documents
Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval
Document language models, query models, and risk minimization for information retrieval
Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval
The normalized recall and related measures
SIGIR '83 Proceedings of the 6th annual international ACM SIGIR conference on Research and development in information retrieval
Information Retrieval
Probabilistic models of information retrieval based on measuring the divergence from randomness
ACM Transactions on Information Systems (TOIS)
Cumulated gain-based evaluation of IR techniques
ACM Transactions on Information Systems (TOIS)
Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval
SIAM Journal on Discrete Mathematics
Retrieval evaluation with incomplete information
Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval
On Event Spaces and Probabilistic Models in Information Retrieval
Information Retrieval
A Markov random field model for term dependencies
Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval
“Irrational” searchers and IR-rational researchers
Journal of the American Society for Information Science and Technology
A decision-theoretic approach to the evaluation of information retrieval systems
Information Processing and Management: an International Journal
Statistical precision of information retrieval evaluation
SIGIR '06 Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval
Rpref: a generalization of Bpref towards graded relevance judgments
SIGIR '06 Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval
Investigating the exhaustivity dimension in content-oriented XML element retrieval evaluation
CIKM '06 Proceedings of the 15th ACM international conference on Information and knowledge management
Estimating average precision with incomplete and imperfect judgments
CIKM '06 Proceedings of the 15th ACM international conference on Information and knowledge management
Evaluating the accuracy of implicit feedback from clicks and query reformulations in Web search
ACM Transactions on Information Systems (TOIS)
On the reliability of information retrieval metrics based on graded relevance
Information Processing and Management: an International Journal - Special issue: AIRS2005: Information retrieval research in Asia
On rank-based effectiveness measures and optimization
Information Retrieval
Computing is a natural science
Communications of the ACM - Creating a science of games
Reliable information retrieval evaluation with incomplete and biased judgements
SIGIR '07 Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval
SIGIR '07 Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval
Strategic system comparisons via targeted relevance judgments
SIGIR '07 Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval
Probability ranking principle via optimal expected rank
SIGIR '07 Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval
Re-examining the effects of adding relevance information in a relevance feedback environment
Information Processing and Management: an International Journal
A probability ranking principle for interactive information retrieval
Information Retrieval
Interpreting TF-IDF term weights as making relevance decisions
ACM Transactions on Information Systems (TOIS)
Score standardization for inter-collection comparison of retrieval systems
Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval
Evaluation measures for preference judgments
Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval
A new interpretation of average precision
Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval
Probabilistic relevance ranking for collaborative filtering
Information Retrieval
On event space and rank equivalence between probabilistic retrieval models
Information Retrieval
Rank-biased precision for measurement of retrieval effectiveness
ACM Transactions on Information Systems (TOIS)
Comparing Scores Intended for Ranking
IEEE Transactions on Knowledge and Data Engineering
Evaluating information retrieval system performance based on multi-grade relevance
ISMIS'08 Proceedings of the 17th international conference on Foundations of intelligent systems
Hi-index | 0.00 |
A new principles framework is presented for retrieval evaluation of ranked outputs. It applies decision theory to model relevance decision preferences and shows that the Probability Ranking Principle (PRP) specifies optimal ranking. It has two new components, namely a probabilistic evaluation model and a general measure of retrieval effectiveness. Its probabilities may be interpreted as subjective or objective ones. Its performance measure is the expected weighted rank which is the weighted average rank of a retrieval list. Starting from this measure, the expected forward rank and some existing retrieval effectiveness measures (e.g., top n precision and discounted cumulative gain) are instantiated using suitable weighting schemes after making certain assumptions. The significance of these instantiations is that the ranking prescribed by PRP is shown to be optimal simultaneously for all these existing performance measures. In addition, the optimal expected weighted rank may be used to normalize the expected weighted rank of retrieval systems for (summary) performance comparison (across different topics) between systems. The framework also extends PRP and our evaluation model to handle graded relevance, thereby generalizing the discussed, existing measures (e.g., top n precision) and probabilistic retrieval models for graded relevance.