A case for interaction: a study of interactive information retrieval behavior and effectiveness
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Do batch and user evaluations give the same results?
SIGIR '00 Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval
Agglomerative clustering of a search engine query log
Proceedings of the sixth ACM SIGKDD international conference on Knowledge discovery and data mining
Information Processing and Management: an International Journal - Special issue on interactivity at the text retrieval conference (TREC)
Why batch and user evaluations do not give the same results
Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval
SIGIR '02 Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval
Re-examining the potential effectiveness of interactive query expansion
Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval
Evaluation of the real and perceived value of automatic and interactive query expansion
Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval
Do clarity scores for queries correlate with user performance?
ADC '04 Proceedings of the 15th Australasian database conference - Volume 27
Exploiting Query Repetition and Regularity in an Adaptive Community-Based Web Search Engine
User Modeling and User-Adapted Interaction
Query association surrogates for Web search: Research Articles
Journal of the American Society for Information Science and Technology
When will information retrieval be "good enough"?
Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval
Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval
User performance versus precision measures for simple search tasks
SIGIR '06 Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval
On ranking the effectiveness of searches
SIGIR '06 Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval
A statistical method for system evaluation using incomplete judgments
SIGIR '06 Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval
Ranking robustness: a novel framework to predict query performance
CIKM '06 Proceedings of the 15th ACM international conference on Information and knowledge management
Examining the effectiveness of real-time query expansion
Information Processing and Management: an International Journal
Studying the use of popular destinations to enhance web search interaction
SIGIR '07 Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval
Query performance prediction in web search environments
SIGIR '07 Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval
Performance prediction using spatial autocorrelation
SIGIR '07 Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval
User adaptation: good results from poor systems
Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval
Improved query difficulty prediction for the web
Proceedings of the 17th ACM conference on Information and knowledge management
A survey of pre-retrieval query performance predictors
Proceedings of the 17th ACM conference on Information and knowledge management
Intentional query suggestion: making user goals more explicit during search
Proceedings of the 2009 workshop on Web Search Click Data
A polyrepresentational approach to interactive query expansion
Proceedings of the 9th ACM/IEEE-CS joint conference on Digital libraries
Efficiency trade-offs in two-tier web search systems
Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval
A comparison of query and term suggestion features for interactive searching
Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval
Predicting Query Performance by Query-Drift Estimation
ICTIR '09 Proceedings of the 2nd International Conference on Theory of Information Retrieval: Advances in Information Retrieval Theory
Search shortcuts: a new approach to the recommendation of queries
Proceedings of the third ACM conference on Recommender systems
Effects of popularity and quality on the usage of query suggestions during information search
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Effective pre-retrieval query performance prediction using similarity and variability evidence
ECIR'08 Proceedings of the IR research, 30th European conference on Advances in information retrieval
Using coherence-based measures to predict query difficulty
ECIR'08 Proceedings of the IR research, 30th European conference on Advances in information retrieval
Predicting Query Performance by Query-Drift Estimation
ACM Transactions on Information Systems (TOIS)
User evaluation of query quality
SIGIR '12 Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval
Query performance prediction for IR
SIGIR '12 Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval
Back to the roots: a probabilistic framework for query-performance prediction
Proceedings of the 21st ACM international conference on Information and knowledge management
Using document-quality measures to predict web-search effectiveness
ECIR'13 Proceedings of the 35th European conference on Advances in Information Retrieval
Understanding Similarity Metrics in Neighbour-based Recommender Systems
Proceedings of the 2013 Conference on the Theory of Information Retrieval
Learning to rank query suggestions for adhoc and diversity search
Information Retrieval
Document Score Distribution Models for Query Performance Inference and Prediction
ACM Transactions on Information Systems (TOIS)
Hi-index | 0.00 |
Query performance prediction methods are usually applied to estimate the retrieval effectiveness of queries, where the evaluation is largely system sided. However, little work has been conducted to understand query performance prediction from the user's perspective. The question we consider is, whether the predictions of query performance that systems make are in line with the predictions that users make. To this aim, we compare the performance ratings users assign to queries with the performance scores estimated by a range of pre-retrieval and post-retrieval query performance predictors. Two studies are presented that explore the relationship between user ratings and system predictions on two levels: (i) the topic level, and, (ii) the query suggestions level. It is shown that when predicting the performance of query suggestions, user ratings were mostly uncorrelated with system predictions. At the topic level though, where a single query is judged for each information need, we observed moderate correlations between user ratings and a subset of system predictions. As query performance prediction methods are often based on intuitions of how users might rate queries, these findings suggest that such methods are not representative of how users actually rate query suggestions and topics. This motivates further research into understanding the rating process engaged by users, and developing models of query performance prediction in order to bridge the divide between systems and users.