Ranking retrieval systems without relevance judgments
Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval
Relevance score normalization for metasearch
Proceedings of the tenth international conference on Information and knowledge management
Linkage and Autocorrelation Cause Feature Selection Bias in Relational Learning
ICML '02 Proceedings of the Nineteenth International Conference on Machine Learning
Using temporal profiles of queries for precision prediction
Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval
Corpus structure, language models, and ad hoc information retrieval
Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval
A study of relevance propagation for web search
Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval
Regularizing ad hoc retrieval scores
Proceedings of the 14th ACM international conference on Information and knowledge management
Minimal test collections for retrieval evaluation
SIGIR '06 Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval
SIGIR '06 Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval
On ranking the effectiveness of searches
SIGIR '06 Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval
A statistical method for system evaluation using incomplete judgments
SIGIR '06 Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval
Precision prediction based on ranked list coherence
Information Retrieval
Ranking robustness: a novel framework to predict query performance
CIKM '06 Proceedings of the 15th ACM international conference on Information and knowledge management
Query hardness estimation using Jensen-Shannon divergence among multiple scoring functions
ECIR'07 Proceedings of the 29th European conference on IR research
Estimating retrieval effectiveness using rank distributions
Proceedings of the 17th ACM conference on Information and knowledge management
From "Identical" to "Similar": Fusing Retrieved Lists Based on Inter-document Similarities
ICTIR '09 Proceedings of the 2nd International Conference on Theory of Information Retrieval: Advances in Information Retrieval Theory
Predicting Query Performance by Query-Drift Estimation
ICTIR '09 Proceedings of the 2nd International Conference on Theory of Information Retrieval: Advances in Information Retrieval Theory
Ranking List Dispersion as a Query Performance Predictor
ICTIR '09 Proceedings of the 2nd International Conference on Theory of Information Retrieval: Advances in Information Retrieval Theory
Relying on topic subsets for system ranking estimation
Proceedings of the 18th ACM conference on Information and knowledge management
Measuring system performance and topic discernment using generalized adaptive-weight mean
Proceedings of the 18th ACM conference on Information and knowledge management
Using statistical decision theory and relevance models for query-performance prediction
Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval
A comparison of user and system query performance predictions
CIKM '10 Proceedings of the 19th ACM international conference on Information and knowledge management
Standard deviation as a query hardness estimator
SPIRE'10 Proceedings of the 17th international conference on String processing and information retrieval
LambdaMerge: merging the results of query reformulations
Proceedings of the fourth ACM international conference on Web search and data mining
Re-ranking search results using an additional retrieved list
Information Retrieval
A unified framework for post-retrieval query-performance prediction
ICTIR'11 Proceedings of the Third international conference on Advances in information retrieval theory
From "identical" to "similar": fusing retrieved lists based on inter-document similarities
Journal of Artificial Intelligence Research
Predicting query performance via classification
ECIR'2010 Proceedings of the 32nd European conference on Advances in Information Retrieval
A case for automatic system evaluation
ECIR'2010 Proceedings of the 32nd European conference on Advances in Information Retrieval
Query performance prediction: evaluation contrasted with effectiveness
ECIR'2010 Proceedings of the 32nd European conference on Advances in Information Retrieval
Predicting Query Performance by Query-Drift Estimation
ACM Transactions on Information Systems (TOIS)
Predicting query performance directly from score distributions
AIRS'11 Proceedings of the 7th Asia conference on Information Retrieval Technology
An uncertainty-aware query selection model for evaluation of IR systems
SIGIR '12 Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval
Query performance prediction for IR
SIGIR '12 Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval
Predicting query performance for fusion-based retrieval
Proceedings of the 21st ACM international conference on Information and knowledge management
Back to the roots: a probabilistic framework for query-performance prediction
Proceedings of the 21st ACM international conference on Information and knowledge management
Query-performance prediction and cluster ranking: two sides of the same coin
Proceedings of the 21st ACM international conference on Information and knowledge management
Using document-quality measures to predict web-search effectiveness
ECIR'13 Proceedings of the 35th European conference on Advances in Information Retrieval
Document Score Distribution Models for Query Performance Inference and Prediction
ACM Transactions on Information Systems (TOIS)
Combining pre-retrieval query quality predictors using genetic programming
Applied Intelligence
Hi-index | 0.00 |
Evaluation of information retrieval systems is one of the core tasks in information retrieval. Problems include the inability to exhaustively label all documents for a topic, generalizability from a small number of topics, and incorporating the variability of retrieval systems. Previous work addresses the evaluation of systems, the ranking of queries by difficulty, and the ranking of individual retrievals by performance. Approaches exist for the case of few and even no relevance judgments. Our focus is on zero-judgment performance prediction of individual retrievals. One common shortcoming of previous techniques is the assumption of uncorrelated document scores and judgments. If documents are embedded in a high-dimensional space (as they often are), we can apply techniques from spatial data analysis to detect correlations between document scores. We find that the low correlation between scores of topically close documents often implies a poor retrieval performance. When compared to a state of the art baseline, we demonstrate that the spatial analysis of retrieval scores provides significantly better prediction performance. These new predictors can also be incorporated with classic predictors to improve performance further. We also describe the first large-scale experiment to evaluate zero-judgment performance prediction for a massive number of retrieval systems over a variety collections in several languages.