Measuring relevance judgements
Information Processing and Management: an International Journal
Determining the effectiveness of retrieval algorithms
Information Processing and Management: an International Journal
The significance of the Cranfield tests on index languages
SIGIR '91 Proceedings of the 14th annual international ACM SIGIR conference on Research and development in information retrieval
Measuring retrieval effectiveness based on user preference of documents
Journal of the American Society for Information Science
21st Annual ACM/SIGIR International Conference on Research and Development in Information Retrieval
Efficient construction of large test collections
Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval
How reliable are the results of large-scale information retrieval experiments?
Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval
Evaluating evaluation measure stability
SIGIR '00 Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval
Variations in relevance judgments and the measurement of retrieval effectiveness
Information Processing and Management: an International Journal
Ranking retrieval systems without relevance judgments
Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval
Evaluation by highly relevant documents
Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval
The effect of topic set size on retrieval experiment error
SIGIR '02 Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval
The Philosophy of Information Retrieval Evaluation
CLEF '01 Revised Papers from the Second Workshop of the Cross-Language Evaluation Forum on Evaluation of Cross-Language Information Retrieval Systems
Automatic ranking of retrieval systems in imperfect environments
Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval
Forming test collections with no system pooling
Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval
Information retrieval system evaluation: effort, sensitivity, and reliability
Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval
Evaluation of resources for question answering evaluation
Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval
When will information retrieval be "good enough"?
Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval
Proceedings of the 14th ACM international conference on Information and knowledge management
Building a reusable test collection for question answering
Journal of the American Society for Information Science and Technology - Research Articles
User performance versus precision measures for simple search tasks
SIGIR '06 Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval
ProbFuse: a probabilistic approach to data fusion
SIGIR '06 Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval
Minimal test collections for retrieval evaluation
SIGIR '06 Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval
Dynamic test collections: measuring search effectiveness on the live web
SIGIR '06 Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval
Statistical precision of information retrieval evaluation
SIGIR '06 Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval
A statistical method for system evaluation using incomplete judgments
SIGIR '06 Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval
First large-scale information retrieval experiments on turkish texts
SIGIR '06 Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval
Rpref: a generalization of Bpref towards graded relevance judgments
SIGIR '06 Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval
Benchmarking image and video retrieval: an overview
MIR '06 Proceedings of the 8th ACM international workshop on Multimedia information retrieval
Benchmarking multimedia search in structured collections
MIR '06 Proceedings of the 8th ACM international workshop on Multimedia information retrieval
Evaluation by comparing result sets in context
CIKM '06 Proceedings of the 15th ACM international conference on Information and knowledge management
Estimating average precision with incomplete and imperfect judgments
CIKM '06 Proceedings of the 15th ACM international conference on Information and knowledge management
Optimisation methods for ranking functions with multiple parameters
CIKM '06 Proceedings of the 15th ACM international conference on Information and knowledge management
Retrieval evaluation with incomplete relevance data: a comparative study of three measures
CIKM '06 Proceedings of the 15th ACM international conference on Information and knowledge management
Creating a test collection for citation-based IR experiments
HLT-NAACL '06 Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics
A machine learning based approach to evaluating retrieval systems
HLT-NAACL '06 Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics
Percent perfect performance (PPP)
Information Processing and Management: an International Journal
On the reliability of information retrieval metrics based on graded relevance
Information Processing and Management: an International Journal - Special issue: AIRS2005: Information retrieval research in Asia
An analysis of two approaches in information retrieval: From frameworks to study designs
Journal of the American Society for Information Science and Technology
Methodologies for Evaluation of Note-Based Music-Retrieval Systems
INFORMS Journal on Computing
The role of information retrieval in answering complex questions
COLING-ACL '06 Proceedings of the COLING/ACL on Main conference poster sessions
ACM Transactions on Software Engineering and Methodology (TOSEM)
Reliable information retrieval evaluation with incomplete and biased judgements
SIGIR '07 Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval
SIGIR '07 Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval
Deconstructing nuggets: the stability and reliability of complex question answering evaluation
SIGIR '07 Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval
On the robustness of relevance measures with incomplete judgments
SIGIR '07 Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval
Strategic system comparisons via targeted relevance judgments
SIGIR '07 Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval
How well does result relevance predict session satisfaction?
SIGIR '07 Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval
A new approach for evaluating query expansion: query-document term mismatch
SIGIR '07 Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval
Power and bias of subset pooling strategies
SIGIR '07 Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval
SIGIR '07 Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval
Expertise modeling for matching papers with reviewers
Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining
Information Processing and Management: an International Journal
Computing block importance for searching on web sites
Proceedings of the sixteenth ACM conference on Conference on information and knowledge management
Just-in-time contextual advertising
Proceedings of the sixteenth ACM conference on Conference on information and knowledge management
Semantic components enhance retrieval of domain-specific documents
Proceedings of the sixteenth ACM conference on Conference on information and knowledge management
Semiautomatic evaluation of retrieval systems using document similarities
Proceedings of the sixteenth ACM conference on Conference on information and knowledge management
Probabilistic data fusion on a large document collection
Artificial Intelligence Review
Fast learning of document ranking functions with the committee perceptron
WSDM '08 Proceedings of the 2008 International Conference on Web Search and Data Mining
Evaluating epistemic uncertainty under incomplete assessments
Information Processing and Management: an International Journal
Assessing multivariate Bernoulli models for information retrieval
ACM Transactions on Information Systems (TOIS)
How robust are multilingual information retrieval systems?
Proceedings of the 2008 ACM symposium on Applied computing
Retrieval sensitivity under training using different measures
Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval
A new rank correlation coefficient for information retrieval
Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval
A simple and efficient sampling method for estimating AP and NDCG
Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval
Novelty and diversity in information retrieval evaluation
Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval
Relevance assessment: are judges exchangeable and does it matter
Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval
Comparing metrics across TREC and NTCIR:: the robustness to pool depth bias
Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval
ArnetMiner: extraction and mining of academic social networks
Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining
Estimating average precision when judgments are incomplete
Knowledge and Information Systems
Keeping a digital library clean: new solutions to old problems
Proceedings of the eighth ACM symposium on Document engineering
Rank-biased precision for measurement of retrieval effectiveness
ACM Transactions on Information Systems (TOIS)
How does clickthrough data reflect retrieval quality?
Proceedings of the 17th ACM conference on Information and knowledge management
Revisiting the relationship between document length and relevance
Proceedings of the 17th ACM conference on Information and knowledge management
Statistical power in retrieval experimentation
Proceedings of the 17th ACM conference on Information and knowledge management
Comparing metrics across TREC and NTCIR: the robustness to system bias
Proceedings of the 17th ACM conference on Information and knowledge management
To swing or not to swing: learning when (not) to advertise
Proceedings of the 17th ACM conference on Information and knowledge management
Using importance flooding to identify interesting networks of criminal activity
Journal of the American Society for Information Science and Technology
Boosting image retrieval through aggregating search results based on visual annotations
MM '08 Proceedings of the 16th ACM international conference on Multimedia
Local search: A guide for the information retrieval practitioner
Information Processing and Management: an International Journal
Effective latent space graph-based re-ranking model with global consistency
Proceedings of the Second ACM International Conference on Web Search and Data Mining
Topic-Level Random Walk through Probabilistic Model
APWeb/WAIM '09 Proceedings of the Joint International Conferences on Advances in Data and Web Management
Using Multiple Query Aspects to Build Test Collections without Human Relevance Judgments
ECIR '09 Proceedings of the 31th European Conference on IR Research on Advances in Information Retrieval
Measuring the Search Effectiveness of a Breadth-First Crawl
ECIR '09 Proceedings of the 31th European Conference on IR Research on Advances in Information Retrieval
A Discriminative Approach to Topic-Based Citation Recommendation
PAKDD '09 Proceedings of the 13th Pacific-Asia Conference on Advances in Knowledge Discovery and Data Mining
Social influence analysis in large-scale networks
Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining
PSkip: estimating relevance ranking quality from web search clickthrough data
Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining
On rank correlation and the distance between rankings
Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval
Score adjustment for correction of pooling bias
Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval
Including summaries in system evaluation
Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval
Searching documentation using text, OCR, and image
Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval
Foundations and Trends in Information Retrieval
Sentiment retrieval using generative models
EMNLP '06 Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing
Building a framework for the probability ranking principle by a family of expected weighted rank
ACM Transactions on Information Systems (TOIS)
Locally contextualized smoothing of language models for sentiment sentence retrieval
Proceedings of the 1st international CIKM workshop on Topic-sentiment analysis for mass opinion
Incorporating user behaviors in new word detection
IJCAI'09 Proceedings of the 21st international jont conference on Artifical intelligence
EMNLP '09 Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 3 - Volume 3
Learning concept importance using a weighted dependence model
Proceedings of the third ACM international conference on Web search and data mining
Measuring the reusability of test collections
Proceedings of the third ACM international conference on Web search and data mining
Click-based evidence for decaying weight distributions in search effectiveness metrics
Information Retrieval
Constructing and mining a semantic-based academic social network
Journal of Intelligent & Fuzzy Systems: Applications in Engineering and Technology - Knowledge integration and management in autonomous systems
Multimodal medical image retrieval: image categorization to improve search precision
Proceedings of the international conference on Multimedia information retrieval
Segmentation of search engine results for effective data-fusion
ECIR'07 Proceedings of the 29th European conference on IR research
A retrieval evaluation methodology for incomplete relevance assessments
ECIR'07 Proceedings of the 29th European conference on IR research
Effective use of semantic structure in XML retrieval
ECIR'07 Proceedings of the 29th European conference on IR research
Generalized distances between rankings
Proceedings of the 19th international conference on World wide web
A mixture model for expert finding
PAKDD'08 Proceedings of the 12th Pacific-Asia conference on Advances in knowledge discovery and data mining
Here or there: preference judgments for relevance
ECIR'08 Proceedings of the IR research, 30th European conference on Advances in information retrieval
Extending probabilistic data fusion using sliding windows
ECIR'08 Proceedings of the IR research, 30th European conference on Advances in information retrieval
Probabilistic document length priors for language models
ECIR'08 Proceedings of the IR research, 30th European conference on Advances in information retrieval
Expert search evaluation by supporting documents
ECIR'08 Proceedings of the IR research, 30th European conference on Advances in information retrieval
Expansão de consulta por pseudo realimentação no modelo TR+ para recuperação de informação
Companion Proceedings of the XIV Brazilian Symposium on Multimedia and the Web
A Wikipedia Matching Approach to Contextual Advertising
World Wide Web
Learning more powerful test statistics for click-based retrieval evaluation
Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval
PRES: a score metric for evaluating recall-oriented information retrieval applications
Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval
Growing related words from seed via user behaviors: a re-ranking based approach
ACLstudent '10 Proceedings of the ACL 2010 Student Research Workshop
Crowdsourcing document relevance assessment with Mechanical Turk
CSLDAMT '10 Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk
A Combination Approach to Web User Profiling
ACM Transactions on Knowledge Discovery from Data (TKDD)
Automatic keyphrase extraction via topic decomposition
EMNLP '10 Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing
CIKM '10 Proceedings of the 19th ACM international conference on Information and knowledge management
Select-the-Best-Ones: A new way to judge relative relevance
Information Processing and Management: an International Journal
Retrieval result presentation and evaluation
KSEM'10 Proceedings of the 4th international conference on Knowledge science, engineering and management
CLEF'10 Proceedings of the 2010 international conference on Multilingual and multimodal information access evaluation: cross-language evaluation forum
Creating a test collection: relevance judgements of cited & non-cited papers
Large Scale Semantic Access to Content (Text, Image, Video, and Sound)
Research methodology in studies of assessor effort for information retrieval evaluation
Large Scale Semantic Access to Content (Text, Image, Video, and Sound)
A comparative analysis of cascade measures for novelty and diversity
Proceedings of the fourth ACM international conference on Web search and data mining
Ranking from pairs and triplets: information quality, evaluation methods and query complexity
Proceedings of the fourth ACM international conference on Web search and data mining
Boiling down information retrieval test collections
RIAO '10 Adaptivity, Personalization and Fusion of Heterogeneous Information
Evaluation effort, reliability and reusability in XML retrieval
Journal of the American Society for Information Science and Technology
Evaluating new search engine configurations with pre-existing judgments and clicks
Proceedings of the 20th international conference on World wide web
The influence of the document ranking in expert search
Information Processing and Management: an International Journal
Evaluation of information retrieval for E-discovery
Artificial Intelligence and Law
Dynamic two-stage image retrieval from large multimodal databases
ECIR'11 Proceedings of the 33rd European conference on Advances in information retrieval
An ontology based model for experts search and ranking
ACIIDS'11 Proceedings of the Third international conference on Intelligent information and database systems - Volume Part II
User Behaviors in Related Word Retrieval and New Word Detection: A Collaborative Perspective
ACM Transactions on Asian Language Information Processing (TALIP)
Web Page Summarization for Just-in-Time Contextual Advertising
ACM Transactions on Intelligent Systems and Technology (TIST)
Discounted cumulative gain and user decision models
SPIRE'11 Proceedings of the 18th international conference on String processing and information retrieval
User effect in evaluating personalized information retrieval systems
EC-TEL'06 Proceedings of the First European conference on Technology Enhanced Learning: innovative Approaches for Learning and Knowledge Sharing
Large-scale validation and analysis of interleaved search evaluation
ACM Transactions on Information Systems (TOIS)
INEX'05 Proceedings of the 4th international conference on Initiative for the Evaluation of XML Retrieval
On effectiveness measures and relevance functions in ranking INEX systems
AIRS'05 Proceedings of the Second Asia conference on Asia Information Retrieval Technology
Intelligent information retrieval tools for police
ISI'06 Proceedings of the 4th IEEE international conference on Intelligence and Security Informatics
A fuzzy ranking approach for improving search results in Turkish as an agglutinative language
Expert Systems with Applications: An International Journal
Bootstrap-Based comparisons of IR metrics for finding one relevant document
AIRS'06 Proceedings of the Third Asia conference on Information Retrieval Technology
Blog opinion retrieval based on topic-opinion mixture model
PAKDD'10 Proceedings of the 14th Pacific-Asia conference on Advances in Knowledge Discovery and Data Mining - Volume Part II
Exploring cost-effective approaches to human evaluation of search engine relevance
ECIR'05 Proceedings of the 27th European conference on Advances in Information Retrieval Research
Scalability influence on retrieval models: an experimental methodology
ECIR'05 Proceedings of the 27th European conference on Advances in Information Retrieval Research
Information retrieval evaluation with partial relevance judgment
BNCOD'06 Proceedings of the 23rd British National Conference on Databases, conference on Flexible and Efficient Information Handling
Turkish information retrieval: past changes future
ADVIS'06 Proceedings of the 4th international conference on Advances in Information Systems
Evaluation of system measures for incomplete relevance judgment in IR
FQAS'06 Proceedings of the 7th international conference on Flexible Query Answering Systems
A classification of IR effectiveness metrics
ECIR'06 Proceedings of the 28th European conference on Advances in Information Retrieval
AMR'10 Proceedings of the 8th international conference on Adaptive Multimedia Retrieval: context, exploration, and fusion
Automated lecture template generation in CORDRA-Based learning object repository
ICWL'11 Proceedings of the 10th international conference on Advances in Web-Based Learning
Modeling and exploiting heterogeneous bibliographic networks for expertise ranking
Proceedings of the 12th ACM/IEEE-CS joint conference on Digital Libraries
PatentMiner: topic-driven patent analysis and mining
Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining
ESWC'12 Proceedings of the 9th international conference on The Semantic Web: research and applications
Combining inverted indices and structured search for ad-hoc object retrieval
SIGIR '12 Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval
Content-based retrieval for heterogeneous domains: domain adaptation by relative aggregation points
SIGIR '12 Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval
Quality through flow and immersion: gamifying crowdsourced relevance assessments
SIGIR '12 Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval
Rhetorical relations for information retrieval
SIGIR '12 Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval
A new performance measure for image retrieval algorithms
International Journal of Information and Communication Technology
Preliminary experiments using subjective logic for the polyrepresentation of information needs
Proceedings of the 4th Information Interaction in Context Symposium
Advertisement selection for online videos
Proceedings of the 18th Brazilian symposium on Multimedia and the web
Recommending citations: translating papers into references
Proceedings of the 21st ACM international conference on Information and knowledge management
Dynamic two-stage image retrieval from large multimedia databases
Information Processing and Management: an International Journal
Measuring the coverage and redundancy of information search services on e-commerce platforms
Electronic Commerce Research and Applications
Wiki3C: exploiting wikipedia for context-aware concept categorization
Proceedings of the sixth ACM international conference on Web search and data mining
Continuous improvement of knowledge management systems using Six Sigma methodology
Robotics and Computer-Integrated Manufacturing
LONET: An interactive search network for intelligent lecture path generation
ACM Transactions on Intelligent Systems and Technology (TIST) - Special section on agent communication, trust in multiagent systems, intelligent tutoring and coaching systems
A Relation Pattern-Driven Probability Model for Related Entity Retrieval
International Journal of Knowledge and Systems Science
An SVD---Bypass latent semantic analysis for image retrieval
MCBR-CDS'12 Proceedings of the Third MICCAI international conference on Medical Content-Based Retrieval for Clinical Decision Support
NE-Rank: A Novel Graph-Based Keyphrase Extraction in Twitter
WI-IAT '12 Proceedings of the The 2012 IEEE/WIC/ACM International Joint Conferences on Web Intelligence and Intelligent Agent Technology - Volume 01
Aggregating evidence from hospital departments to improve medical records search
ECIR'13 Proceedings of the 35th European conference on Advances in Information Retrieval
ECIR'13 Proceedings of the 35th European conference on Advances in Information Retrieval
Automatic tag recommendation for metadata annotation using probabilistic topic modeling
Proceedings of the 13th ACM/IEEE-CS joint conference on Digital libraries
Journal of Web Engineering
Click model-based information retrieval metrics
Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval
A mutual information-based framework for the analysis of information retrieval systems
Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval
A general evaluation measure for document organization tasks
Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval
Preference based evaluation measures for novelty and diversity
Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval
Building a web test collection using social media
Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval
Know your personalization: learning topic level personalization in online services
Proceedings of the 22nd international conference on World Wide Web
Inferring conceptual relationships to improve medical records search
Proceedings of the 10th Conference on Open Research Areas in Information Retrieval
"Who's out there?": identifying and ranking lurkers in social networks
Proceedings of the 2013 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining
Query-Performance Prediction Using Minimal Relevance Feedback
Proceedings of the 2013 Conference on the Theory of Information Retrieval
Learning to handle negated language in medical records search
Proceedings of the 22nd ACM international conference on Conference on information & knowledge management
A new statistical strategy for pooling: ELI
Information Processing Letters
Information quality measurement of medical encoding support based on usability
Computer Methods and Programs in Biomedicine
Bridging memory-based collaborative filtering and text retrieval
Information Retrieval
Evaluation as a service for information retrieval
ACM SIGIR Forum
Evaluation in Music Information Retrieval
Journal of Intelligent Information Systems
Multimodal retrieval with relevance feedback based on genetic programming
Multimedia Tools and Applications
Semantic concept-enriched dependence model for medical information retrieval
Journal of Biomedical Informatics
Hi-index | 0.00 |
This paper examines whether the Cranfield evaluation methodology is robust to gross violations of the completeness assumption (i.e., the assumption that all relevant documents within a test collection have been identified and are present in the collection). We show that current evaluation measures are not robust to substantially incomplete relevance judgments. A new measure is introduced that is both highly correlated with existing measures when complete judgments are available and more robust to incomplete judgment sets. This finding suggests that substantially larger or dynamic test collections built using current pooling practices should be viable laboratory tools, despite the fact that the relevance information will be incomplete and imperfect.