Combining multiple evidence from different properties of weighting schemes
SIGIR '95 Proceedings of the 18th annual international ACM SIGIR conference on Research and development in information retrieval
Variations in relevance assessments and the measurement of retrieval effectiveness
Journal of the American Society for Information Science - Special issue: evaluation of information retrieval systems
Analyses of multiple evidence combination
Proceedings of the 20th annual international ACM SIGIR conference on Research and development in information retrieval
How reliable are the results of large-scale information retrieval experiments?
Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval
Estimating precision by random sampling (poster abstract)
Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval
Evaluating evaluation measure stability
SIGIR '00 Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval
Variations in relevance judgments and the measurement of retrieval effectiveness
Information Processing and Management: an International Journal
Rank aggregation methods for the Web
Proceedings of the 10th international conference on World Wide Web
Ranking retrieval systems without relevance judgments
Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval
Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval
Building efficient and effective metasearch engines
ACM Computing Surveys (CSUR)
Using SPSS to Solve Statistical Problems: A Self Instruction Guide
Using SPSS to Solve Statistical Problems: A Self Instruction Guide
Advances in Informational Retrieval: Recent Research from the Center for Intelligent Information Retrieval
SIGIR '02 Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval
Assessing bias in search engines
Information Processing and Management: an International Journal
Introduction to Modern Information Retrieval
Introduction to Modern Information Retrieval
Condorcet fusion for improved retrieval
Proceedings of the eleventh international conference on Information and knowledge management
The Philosophy of Information Retrieval Evaluation
CLEF '01 Revised Papers from the Second Workshop of the Cross-Language Evaluation Forum on Evaluation of Cross-Language Information Retrieval Systems
Automatic ranking of retrieval systems in imperfect environments
Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval
Methods for ranking information retrieval systems without relevance judgments
Proceedings of the 2003 ACM symposium on Applied computing
Proceedings of the 2003 ACM symposium on Applied computing
Using titles and category names from editor-driven taxonomies for automatic evaluation
CIKM '03 Proceedings of the twelfth international conference on Information and knowledge management
Automatic performance evaluation of web search engines
Information Processing and Management: an International Journal
Scaling IR-system evaluation using term relevance sets
Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval
Journal of the American Society for Information Science and Technology
Representing and aggregating conflicting beliefs
Journal of Artificial Intelligence Research
Improving high accuracy retrieval by eliminating the uneven correlation effect in data fusion
Journal of the American Society for Information Science and Technology
A machine learning based approach to evaluating retrieval systems
HLT-NAACL '06 Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics
Repeatable evaluation of search services in dynamic environments
ACM Transactions on Information Systems (TOIS)
Incremental cluster-based retrieval using compressed cluster-skipping inverted files
ACM Transactions on Information Systems (TOIS)
Assigning appropriate weights for the linear combination data fusion method in information retrieval
Information Processing and Management: an International Journal
Concept, content and the convict
MM '09 Proceedings of the 17th ACM international conference on Multimedia
Relying on topic subsets for system ranking estimation
Proceedings of the 18th ACM conference on Information and knowledge management
On the Selection of the Best Retrieval Result Per Query ---An Alternative Approach to Data Fusion---
FQAS '09 Proceedings of the 8th International Conference on Flexible Query Answering Systems
New event detection and topic tracking in Turkish
Journal of the American Society for Information Science and Technology
Web search solved?: all result rankings the same?
CIKM '10 Proceedings of the 19th ACM international conference on Information and knowledge management
Aspects and analysis of patent test collections
PaIR '10 Proceedings of the 3rd international workshop on Patent information retrieval
Global ranking via data fusion
COLING '10 Proceedings of the 23rd International Conference on Computational Linguistics: Posters
Using clustering to improve retrieval evaluation without relevance judgments
COLING '10 Proceedings of the 23rd International Conference on Computational Linguistics: Posters
Beyond Travel & Tourism competitiveness ranking using DEA, GST, ANN and Borda count
Expert Systems with Applications: An International Journal
An overview of Web search evaluation methods
Computers and Electrical Engineering
A case for automatic system evaluation
ECIR'2010 Proceedings of the 32nd European conference on Advances in Information Retrieval
Multimodal information spaces for content-based image retrieval
FDIA'09 Proceedings of the Third BCS-IRSG conference on Future Directions in Information Access
Improving BCI performance after classification
Proceedings of the 14th ACM international conference on Multimodal interaction
The weighted Condorcet fusion in information retrieval
Information Processing and Management: an International Journal
Pattern Recognition Letters
Hi-index | 0.00 |
Measuring effectiveness of information retrieval (IR) systems is essential for research and development and for monitoring search quality in dynamic environments. In this study, we employ new methods for automatic ranking of retrieval systems. In these methods, we merge the retrieval results of multiple systems using various data fusion algorithms, use the top-ranked documents in the merged result as the "(pseudo) relevant documents," and employ these documents to evaluate and rank the systems. Experiments using Text REtrieval Conference (TREC) data provide statistically significant strong correlations with human-based assessments of the same systems. We hypothesize that the selection of systems that would return documents different from the majority could eliminate the ordinary systems from data fusion and provide better discrimination among the documents and systems. This could improve the effectiveness of automatic ranking. Based on this intuition, we introduce a new method for the selection of systems to be used for data fusion. For this purpose, we use the bias concept that measures the deviation of a system from the norm or majority and employ the systems with higher bias in the data fusion process. This approach provides even higher correlations with the human-based results. We demonstrate that our approach outperforms the previously proposed automatic ranking methods.