Measures of relative relevance and ranked half-life: performance indicators for interactive IR
Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval
Text retrieval and filtering: analytic models of performance
Text retrieval and filtering: analytic models of performance
A technique for measuring the relative size and overlap of public Web search engines
WWW7 Proceedings of the seventh international conference on World Wide Web 7
Finding information on the World Wide Web: the retrieval effectiveness of search engines
Information Processing and Management: an International Journal
IR evaluation methods for retrieving highly relevant documents
SIGIR '00 Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval
Rank-preserving two-level caching for scalable search engines
Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval
Information Retrieval
Automatic evaluation of world wide web search services
SIGIR '02 Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval
Cumulated gain-based evaluation of IR techniques
ACM Transactions on Information Systems (TOIS)
Measuring Search Engine Quality
Information Retrieval
Precision Evaluation of Search Engines
World Wide Web
SODA '03 Proceedings of the fourteenth annual ACM-SIAM symposium on Discrete algorithms
ACM SIGIR Forum
Automatic performance evaluation of web search engines
Information Processing and Management: an International Journal
Search engine coverage bias: evidence and possible causes
Information Processing and Management: an International Journal
A subjective measure of web search quality
Information Sciences—Informatics and Computer Science: An International Journal
A framework for determining necessary query set sizes to evaluate web search effectiveness
WWW '05 Special interest tracks and posters of the 14th international conference on World Wide Web
Information Processing and Management: an International Journal
When will information retrieval be "good enough"?
Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval
Comparing rankings of search results on the web
Information Processing and Management: an International Journal - Special issue: Infometrics
Automatic ranking of information retrieval systems using data fusion
Information Processing and Management: an International Journal
Methods for comparing rankings of search engine results
Computer Networks: The International Journal of Computer and Telecommunications Networking - Web dynamics
A study of results overlap and uniqueness among major web search engines
Information Processing and Management: an International Journal
Optimisation methods for ranking functions with multiple parameters
CIKM '06 Proceedings of the 15th ACM international conference on Information and knowledge management
Defining a session on Web search engines: Research Articles
Journal of the American Society for Information Science and Technology
SIAM Journal on Discrete Mathematics
The relationship between IR effectiveness measures and user satisfaction
SIGIR '07 Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval
User rankings of search engine results
Journal of the American Society for Information Science and Technology
Entropy of search logs: how hard is search? with personalization? with backoff?
WSDM '08 Proceedings of the 2008 International Conference on Web Search and Data Mining
User adaptation: good results from poor systems
Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval
A new rank correlation coefficient for information retrieval
Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval
Relevance assessment: are judges exchangeable and does it matter
Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval
Quantitative comparisons of search engine results
Journal of the American Society for Information Science and Technology
Rank-biased precision for measurement of retrieval effectiveness
ACM Transactions on Information Systems (TOIS)
A three-year study on the freshness of web search engine databases
Journal of Information Science
Improving Search Performance: A Lesson Learned from Evaluating Search Engines Using Thai Queries
IEICE - Transactions on Information and Systems
What users see - Structures in search engine results pages
Information Sciences: an International Journal
A survey on session detection methods in query logs and a proposal for future evaluation
Information Sciences: an International Journal
On rank correlation and the distance between rankings
Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval
Expected reciprocal rank for graded relevance
Proceedings of the 18th ACM conference on Information and knowledge management
Relevance measurement on chinese search results
HCI'07 Proceedings of the 12th international conference on Human-computer interaction: applications and services
Generalized distances between rankings
Proceedings of the 19th international conference on World wide web
Coverage and timeliness analysis of search engines with webpage monitoring results
WISE'07 Proceedings of the 8th international conference on Web information systems engineering
Search intention analysis for user-centered adaptive visualizations
UAHCI'11 Proceedings of the 6th international conference on Universal access in human-computer interaction: design for all and eInclusion - Volume Part I
Reranking search results for sparse queries
Proceedings of the 20th ACM international conference on Information and knowledge management
Google, bing and a new perspective on ranking similarity
Proceedings of the 20th ACM international conference on Information and knowledge management
Beyond precision@10: clustering the long tail of web search results
Proceedings of the 20th ACM international conference on Information and knowledge management
Answers, not links: extracting tips from yahoo! answers to address how-to web queries
Proceedings of the fifth ACM international conference on Web search and data mining
Can click patterns across user's query logs predict answers to definition questions?
EACL '12 Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics
Collaborative ranking: improving the relevance for tail queries
Proceedings of the 21st ACM international conference on Information and knowledge management
Characterizing web search queries that match very few or no results
Proceedings of the 21st ACM international conference on Information and knowledge management
Improving the performance of the reinforcement learning model for answering complex questions
Proceedings of the 21st ACM international conference on Information and knowledge management
Analyzing, Detecting, and Exploiting Sentiment in Web Queries
ACM Transactions on the Web (TWEB)
The cross-lingual lexical substitution task
Language Resources and Evaluation
| Hi-index | 0.00 |
The objective of this work is to derive quantitative statements about what fraction of web search queries issued to the state-of-the-art commercial search engines lead to excellent results or, on the contrary, poor results. To be able to make such statements in an automated way, we propose a new measure that is based on lower and upper bound analysis over the standard relevance measures. Moreover, we extend this measure to carry out comparisons between competing search engines by introducing the concept of disruptive sets, which we use to estimate the degree to which a search engine solves queries that are not solved by its competitors. We report empirical results on a large editorial evaluation of the three largest search engines in the US market.