An evaluation of retrieval effectiveness for a full-text document-retrieval system
Communications of the ACM
Information Processing and Management: an International Journal - The Potential for Improvments in Commerical Document Retrieval Systems
Measuring relevance judgements
Information Processing and Management: an International Journal
A re-examination of relevance: toward a dynamic, situational definition
Information Processing and Management: an International Journal
The use of term position devices in ranked output experiments
Journal of Documentation
Inference networks for document retrieval
Inference networks for document retrieval
Information retrieval evaluation in practice: a case study approach
Information Processing and Management: an International Journal
User-defined relevance criteria: an exploratory study
Journal of the American Society for Information Science - Special issue: relevance research
Combining automatic and manual index representations in probabilistic retrieval
Journal of the American Society for Information Science
An evaluation of interactive Boolean and natural language searching with an online medical textbook
Journal of the American Society for Information Science
A cognitive model of document use during a research project. Study I. document selection
Journal of the American Society for Information Science
Users' criteria for relevance evaluation: a cross-situational comparison
Information Processing and Management: an International Journal
The impact of query structure and query expansion on retrieval performance
Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval
Text retrieval and filtering: analytic models of performance
Text retrieval and filtering: analytic models of performance
Towards the identification of the optimal number of relevance categories
Journal of the American Society for Information Science
From highly relevant to not relevant: examining different regions of relevance
Information Processing and Management: an International Journal
IR evaluation methods for retrieving highly relevant documents
SIGIR '00 Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval
Information Processing and Management: an International Journal
Evaluation by highly relevant documents
Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval
Liberal relevance criteria of TREC -: counting on negligible documents?
SIGIR '02 Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval
Cumulated gain-based evaluation of IR techniques
ACM Transactions on Information Systems (TOIS)
XML retrieval: what to retrieve?
Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval
A report on the first year of the INitiative for the evaluation of XML retrieval (INEX'02)
Journal of the American Society for Information Science and Technology
The overlap problem in content-oriented XML retrieval evaluation
Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval
The SST method: a tool for analysing web information search processes
Information Processing and Management: an International Journal
Providing consistent and exhaustive relevance assessments for XML retrieval evaluation
Proceedings of the thirteenth ACM international conference on Information and knowledge management
Stemming and lemmatization in the clustering of finnish text documents
Proceedings of the thirteenth ACM international conference on Information and knowledge management
The influence of relevance levels on the effectiveness of interactive information retrieval
Journal of the American Society for Information Science and Technology
Information Retrieval
A general matrix framework for modelling information retrieval
Information Processing and Management: an International Journal - Special issue: Formal methods for information retrieval
Evaluation in (XML) information retrieval: expected precision-recall with user modelling (EPRUM)
SIGIR '06 Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval
Rpref: a generalization of Bpref towards graded relevance judgments
SIGIR '06 Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval
SIGIR '06 Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval
eXtended cumulated gain measures for the evaluation of content-oriented XML retrieval
ACM Transactions on Information Systems (TOIS)
Precision recall with user modeling (PRUM): Application to structured information retrieval
ACM Transactions on Information Systems (TOIS)
Journal of the American Society for Information Science and Technology
Journal of the American Society for Information Science and Technology
Answering bounded continuous search queries in the world wide web
Proceedings of the 16th international conference on World Wide Web
Evaluating XML retrieval effectiveness at INEX
ACM SIGIR Forum
Evaluating relevant in context: document retrieval with a twist
SIGIR '07 Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval
Evaluation of a mobile information system in context
Information Processing and Management: an International Journal
Efficient search ranking in social networks
Proceedings of the sixteenth ACM conference on Conference on information and knowledge management
Novelty and topicality in interactive information retrieval
Journal of the American Society for Information Science and Technology
Sound and complete relevance assessment for XML retrieval
ACM Transactions on Information Systems (TOIS)
Overview of the INEX 2007 Ad Hoc Track
Focused Access to XML Documents
Focused Access to XML Documents
Overview of INEX 2007 Link the Wiki Track
Focused Access to XML Documents
Assessing thesaurus-based annotations for semantic search applications
International Journal of Metadata, Semantics and Ontologies
Local search: A guide for the information retrieval practitioner
Information Processing and Management: an International Journal
Methods for Evaluating Interactive Information Retrieval Systems with Users
Foundations and Trends in Information Retrieval
A general matrix framework for modelling Information Retrieval
Information Processing and Management: an International Journal - Special issue: Formal methods for information retrieval
Aspects of 'relevance' in the alignment of curriculum with educational standards
Information Processing and Management: an International Journal
Relevance measurement on chinese search results
HCI'07 Proceedings of the 12th international conference on Human-computer interaction: applications and services
Optimization of bounded continuous search queries based on ranking distributions
WISE'07 Proceedings of the 8th international conference on Web information systems engineering
INEX 2002-2006: understanding XML retrieval evaluation
DELOS'07 Proceedings of the 1st international conference on Digital libraries: research and development
Two variations on ontology alignment evaluation: methodological issues
ESWC'08 Proceedings of the 5th European semantic web conference on The semantic web: research and applications
Determining relevance of imprecise temporal intervals for cultural heritage information retrieval
International Journal of Human-Computer Studies
Thesaurus enrichment for query expansion in audiovisual archives
Multimedia Tools and Applications
Extending average precision to graded relevance judgments
Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval
Assessors' search result satisfaction associated with relevance in a scientific domain
Proceedings of the third symposium on Information interaction in context
Expected reading effort in focused retrieval evaluation
Information Retrieval
Overview of the INEX 2009 ad hoc track
INEX'09 Proceedings of the Focused retrieval and evaluation, and 8th international conference on Initiative for the evaluation of XML retrieval
Overview of VideoCLEF 2009: new perspectives on speech-based multimedia content enrichment
CLEF'09 Proceedings of the 10th international conference on Cross-language evaluation forum: multimedia experiments
A cocktail approach to the VideoCLEF'09 linking task
CLEF'09 Proceedings of the 10th international conference on Cross-language evaluation forum: multimedia experiments
CLEF'09 Proceedings of the 10th cross-language evaluation forum conference on Multilingual information access evaluation: text retrieval experiments
ACM Transactions on Speech and Language Processing (TSLP)
Overview of the INEX 2010 ad hoc track
INEX'10 Proceedings of the 9th international conference on Initiative for the evaluation of XML retrieval: comparative evaluation of focused retrieval
The potential benefit of focused retrieval in relevant-in-context task
INEX'10 Proceedings of the 9th international conference on Initiative for the evaluation of XML retrieval: comparative evaluation of focused retrieval
INEX'05 Proceedings of the 4th international conference on Initiative for the Evaluation of XML Retrieval
On effectiveness measures and relevance functions in ranking INEX systems
AIRS'05 Proceedings of the Second Asia conference on Asia Information Retrieval Technology
Evaluating scalability in information retrieval with multigraded relevance
AIRS'06 Proceedings of the Third Asia conference on Information Retrieval Technology
ECIR'05 Proceedings of the 27th European conference on Advances in Information Retrieval Research
Dictionary-based CLIR loses highly relevant documents
ECIR'05 Proceedings of the 27th European conference on Advances in Information Retrieval Research
Contexts of relevance for information retrieval system design
CoLIS'05 Proceedings of the 5th international conference on Context: conceptions of Library and Information Sciences
Searching for relevance in the relevance of search
CoLIS'05 Proceedings of the 5th international conference on Context: conceptions of Library and Information Sciences
Context matters: an analysis of assessments of XML documents
CoLIS'05 Proceedings of the 5th international conference on Context: conceptions of Library and Information Sciences
Reliability tests for the XCG and inex-2002 metrics
INEX'04 Proceedings of the Third international conference on Initiative for the Evaluation of XML Retrieval
ECIR'06 Proceedings of the 28th European conference on Advances in Information Retrieval
New metrics for meaningful evaluation of informally structured speech retrieval
ECIR'12 Proceedings of the 34th European conference on Advances in Information Retrieval
Overview of the CLEF-2006 cross-language speech retrieval track
CLEF'06 Proceedings of the 7th international conference on Cross-Language Evaluation Forum: evaluation of multilingual and multi-modal information retrieval
Cumulated relative position: a metric for ranking evaluation
CLEF'12 Proceedings of the Third international conference on Information Access Evaluation: multilinguality, multimodality, and visual analytics
Ranking Tagged Resources Using Social Semantic Relevance
International Journal of Information Retrieval Research
One size does not fit all: multi-granularity search of web forums
Proceedings of the 22nd ACM international conference on Conference on information & knowledge management
Exploiting user disagreement for web search evaluation: an experimental approach
Proceedings of the 7th ACM international conference on Web search and data mining
Hi-index | 0.00 |
This article proposes evaluation methods based on the use of nondichotomous relevance judgements in IR experiments. It is argued that evaluation methods should credit IR methods for their ability to retrieve highly relevant documents. This is desirable from the user point of view in modern large IR environments. The proposed methods are (1) a novel application of P-R curves and average precision computations based on separate recall bases for documents of different degrees of relevance, and (2) generalized recall and precision based directly on multiple grade relevance assessments (i.e., not dichotomizing the assessments). We demonstrate the use of the traditional and the novel evaluation measures in a case study on the effectiveness of query types, based on combinations of query structures and expansion, in retrieving documents of various degrees of relevance. The test was run with a best match retrieval system (InQuery1) in a text database consisting of newspaper articles. To gain insight into the retrieval process, one should use both graded relevance assessments and effectiveness measures that enable one to observe the differences, if any, between retrieval methods in retrieving documents of different levels of relevance. In modern times of information overload, one should pay attention, in particular, to the capability of retrieval methods retrieving highly relevant documents.