An evaluation of retrieval effectiveness for a full-text document-retrieval system
Communications of the ACM
Annual review of information science and technology, vol. 22
Automatic text processing: the transformation, analysis, and retrieval of information by computer
Automatic text processing: the transformation, analysis, and retrieval of information by computer
Integration of user profiles: models and experiments in information retrieval
Information Processing and Management: an International Journal
The use of term position devices in ranked output experiments
Journal of Documentation
Inference networks for document retrieval
Inference networks for document retrieval
The pragmatics of information retrieval experimentation, revisited
Information Processing and Management: an International Journal - Special issue on evaluation issues in information retrieval
Presenting results of experimental retrieval comparisons
Information Processing and Management: an International Journal - Special issue on evaluation issues in information retrieval
Using statistical testing in the evaluation of retrieval experiments
SIGIR '93 Proceedings of the 16th annual international ACM SIGIR conference on Research and development in information retrieval
Combining automatic and manual index representations in probabilistic retrieval
Journal of the American Society for Information Science
An evaluation of interactive Boolean and natural language searching with an online medical textbook
Journal of the American Society for Information Science
Information retrieval: a health care perspective
Information retrieval: a health care perspective
WordNet: a lexical database for English
Communications of the ACM
Query expansion using local and global document analysis
SIGIR '96 Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval
A deductive data model for query expansion
SIGIR '96 Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval
Information storage and retrieval
Information storage and retrieval
The impact of query structure and query expansion on retrieval performance
Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval
Measures of relative relevance and ranked half-life: performance indicators for interactive IR
Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval
Text retrieval and filtering: analytic models of performance
Text retrieval and filtering: analytic models of performance
The rise of ontologies or the reinvention of classification
Journal of the American Society for Information Science - Special issue on the 50th anniversary of the Journal of The American Society for Information Science: part 2: paradigms, models and methods of information science
Bringing order to the Web: automatically categorizing search results
Proceedings of the SIGCHI conference on Human Factors in Computing Systems
IR evaluation methods for retrieving highly relevant documents
SIGIR '00 Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval
Information Processing and Management: an International Journal
Personal ontologies for web navigation
Proceedings of the ninth international conference on Information and knowledge management
Introduction to Modern Information Retrieval
Introduction to Modern Information Retrieval
ExpansionTool: Concept-Based Query Expansion and Construction
Information Retrieval
Hi-index | 0.00 |
This chapter discusses evaluation methods based on the use of nondichotomous relevance judgments in information retrieval (IR) experiments. It is argued that evaluation methods should credit IR methods for their ability to retrieve highly relevant documents. This is desirable from the user's point of view in modern large IR environments. The proposed methods are (1) a novel application of P-R curves and average precision computations based on separate recall bases for documents of different degrees of relevance, and (2) two novel measures computing the cumulative gain the user obtains by examining the retrieval result up to a given ranked position. We then demonstrate the use of these evaluation methods in a case study on the effectiveness of query types, based on combinations of query structures and expansion, in retrieving documents of various degrees of relevance. Query expansion is based on concepts, which are selected from a conceptual model, and then expanded by semantic relationships given in the model. The test is run with a best-match retrieval system (InQuery) in a text database consisting of newspaper articles. The case study indicates the usability of domain-dependent conceptual models in query expansion for IR. The results show that expanded queries with a strong query structure are most effective in retrieving highly relevant documents. The differences between the query types are practically essential and statistically significant. More generally, the novel evaluation methods and the case demonstrate that nondichotomous relevance assessments are applicable in IR experiments and allow harder testing of IR methods. Proposed methods are user-oriented because users' benefits and efforts--highly relevant documents and number of documents to be examined-- are taken into account.