Variations in relevance assessments and the measurement of retrieval effectiveness
Journal of the American Society for Information Science - Special issue: evaluation of information retrieval systems
The Cranfield tests on index language devices
Readings in information retrieval
Advantages of query biased summaries in information retrieval
Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval
How reliable are the results of large-scale information retrieval experiments?
Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval
Do batch and user evaluations give the same results?
SIGIR '00 Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval
Evaluating evaluation measure stability
SIGIR '00 Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval
Variations in relevance judgments and the measurement of retrieval effectiveness
Information Processing and Management: an International Journal
Why batch and user evaluations do not give the same results
Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval
The Philosophy of Information Retrieval Evaluation
CLEF '01 Revised Papers from the Second Workshop of the Cross-Language Evaluation Forum on Evaluation of Cross-Language Information Retrieval Systems
ACM SIGIR Forum
Engineering a multi-purpose test collection for web retrieval experiments
Information Processing and Management: an International Journal
Retrieval evaluation with incomplete information
Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval
Accurately interpreting clickthrough data as implicit feedback
Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval
When will information retrieval be "good enough"?
Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval
TREC: Experiment and Evaluation in Information Retrieval (Digital Libraries and Electronic Publishing)
An analysis of two approaches in information retrieval: From frameworks to study designs
Journal of the American Society for Information Science and Technology
SIGIR '07 Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval
How well does result relevance predict session satisfaction?
SIGIR '07 Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval
SIGIR '07 Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval
Semantic components enhance retrieval of domain-specific documents
Proceedings of the sixteenth ACM conference on Conference on information and knowledge management
Hierarchical summarization for delivering information to mobile devices
Information Processing and Management: an International Journal
A probability ranking principle for interactive information retrieval
Information Retrieval
Learning to learn implicit queries from gaze patterns
Proceedings of the 25th international conference on Machine learning
Learning diverse rankings with multi-armed bandits
Proceedings of the 25th international conference on Machine learning
How do users find things with PubMed?: towards automatic utility evaluation with user simulations
Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval
Score standardization for inter-collection comparison of retrieval systems
Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval
The good and the bad system: does the test collection predict users' effectiveness?
Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval
User adaptation: good results from poor systems
Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval
Relevance judgments between TREC and Non-TREC assessors
Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval
Relevance thresholds in system evaluations
Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval
Precision-at-ten considered redundant
Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval
Some(what) grand challenges for information retrieval
ACM SIGIR Forum
Experiences evaluating personal metasearch
Proceedings of the second international symposium on Information interaction in context
Toward automatic facet analysis and need negotiation: Lessons from mediated search
ACM Transactions on Information Systems (TOIS)
How does clickthrough data reflect retrieval quality?
Proceedings of the 17th ACM conference on Information and knowledge management
Multiple coordinated views for searching and navigating Web content repositories
Information Sciences: an International Journal
Including summaries in system evaluation
Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval
Evaluating web search using task completion time
Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval
Explaining User Performance in Information Retrieval: Challenges to IR Evaluation
ICTIR '09 Proceedings of the 2nd International Conference on Theory of Information Retrieval: Advances in Information Retrieval Theory
Methods for Evaluating Interactive Information Retrieval Systems with Users
Foundations and Trends in Information Retrieval
Can eyes reveal interest? Implicit queries from gaze patterns
User Modeling and User-Adapted Interaction
The influence of the document ranking in expert search
Proceedings of the 18th ACM conference on Information and knowledge management
Metric and Relevance Mismatch in Retrieval Evaluation
AIRS '09 Proceedings of the 5th Asia Information Retrieval Symposium on Information Retrieval Technology
Beyond DCG: user behavior as a predictor of a successful search
Proceedings of the third ACM international conference on Web search and data mining
ACM Transactions on Information Systems (TOIS)
Adoption of translation support technologies in a multilingual work environment
IWIC'07 Proceedings of the 1st international conference on Intercultural collaboration
Proceedings of the 19th international conference on World wide web
Anonymizing user profiles for personalized web search
Proceedings of the 19th international conference on World wide web
Using clicks as implicit judgments: expectations versus observations
ECIR'08 Proceedings of the IR research, 30th European conference on Advances in information retrieval
Investigating the effectiveness of clickthrough data for document reordering
ECIR'08 Proceedings of the IR research, 30th European conference on Advances in information retrieval
A new automated information retrieval system by using intelligent mobile agent
AIKED'10 Proceedings of the 9th WSEAS international conference on Artificial intelligence, knowledge engineering and data bases
Tightly coupled views for navigating content repositories
Companion Proceedings of the XIV Brazilian Symposium on Multimedia and the Web
A review of factors influencing user satisfaction in information retrieval
Journal of the American Society for Information Science and Technology
The good, the bad, and the random: an eye-tracking study of ad quality in web search
Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval
Do user preferences and evaluation measures line up?
Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval
Human performance and retrieval precision revisited
Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval
Comparing the sensitivity of information retrieval metrics
Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval
Using query context models to construct topical search engines
Proceedings of the third symposium on Information interaction in context
Evaluating search systems using result page context
Proceedings of the third symposium on Information interaction in context
On the potential search effectiveness of MeSH (medical subject headings) terms
Proceedings of the third symposium on Information interaction in context
A comparison of user and system query performance predictions
CIKM '10 Proceedings of the 19th ACM international conference on Information and knowledge management
Surfing on artistic documents with visually assisted tagging
Proceedings of the international conference on Multimedia
MICCAI'10 Proceedings of the 13th international conference on Medical image computing and computer-assisted intervention: Part II
The influence of the document ranking in expert search
Information Processing and Management: an International Journal
Evaluation of information retrieval for E-discovery
Artificial Intelligence and Law
Find it if you can: a game for modeling different types of web search success using interaction data
Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval
What deliberately degrading search quality tells us about discount functions
Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval
The effect of user characteristics on search effectiveness in information retrieval
Information Processing and Management: an International Journal
BPM'11 Proceedings of the 9th international conference on Business process management
Query-feature graphs: bridging user vocabulary and system functionality
Proceedings of the 24th annual ACM symposium on User interface software and technology
TOPSIG: topology preserving document signatures
Proceedings of the 20th ACM international conference on Information and knowledge management
Simulating simple user behavior for system effectiveness evaluation
Proceedings of the 20th ACM international conference on Information and knowledge management
Software engineers' information behaviour and implicit relevance indicators
International Journal of Knowledge and Web Intelligence
IR research: systems, interaction, evaluation and theories
ACM SIGIR Forum
Large-scale validation and analysis of interleaved search evaluation
ACM Transactions on Information Systems (TOIS)
Task-specific information retrieval systems for software engineers
Journal of Computer and System Sciences
Time drives interaction: simulating sessions in diverse searching environments
SIGIR '12 Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval
Amount of invested mental effort (AIME) in online searching
Information Processing and Management: an International Journal
Indexing and retrieval of medical resources for a telemedical platform
ITIB'12 Proceedings of the Third international conference on Information Technologies in Biomedicine
Mining millions of reviews: a technique to rank products based on importance of reviews
Proceedings of the 13th International Conference on Electronic Commerce
On the role of novelty for search result diversification
Information Retrieval
Semi-supervised spectral hashing for fast similarity search
Neurocomputing
Models and metrics: IR evaluation as a user process
Proceedings of the Seventeenth Australasian Document Computing Symposium
An Investigation of User Behaviour Consistency for Context-Aware Information Retrieval Systems
International Journal of Advanced Pervasive and Ubiquitous Computing
Users versus models: what observation tells us about effectiveness metrics
Proceedings of the 22nd ACM international conference on Conference on information & knowledge management
Choices in batch information retrieval evaluation
Proceedings of the 18th Australasian Document Computing Symposium
IJCAI'13 Proceedings of the Twenty-Third international joint conference on Artificial Intelligence
Latent dirichlet allocation based diversified retrieval for e-commerce search
Proceedings of the 7th ACM international conference on Web search and data mining
Clustering results of image searches by annotations and visual features
Telematics and Informatics
CFinder: An intelligent key concept finder from text for ontology development
Expert Systems with Applications: An International Journal
Hi-index | 0.00 |
Several recent studies have demonstrated that the type of improvements in information retrieval system effectiveness reported in forums such as SIGIR and TREC do not translate into a benefit for users. Two of the studies used an instance recall task, and a third used a question answering task, so perhaps it is unsurprising that the precision based measures of IR system effectiveness on one-shot query evaluation do not correlate with user performance on these tasks. In this study, we evaluate two different information retrieval tasks on TREC Web-track data: a precision-based user task, measured by the length of time that users need to find a single document that is relevant to a TREC topic; and, a simple recall-based task, represented by the total number of relevant documents that users can identify within five minutes. Users employ search engines with controlled mean average precision (MAP) of between 55% and 95%. Our results show that there is no significant relationship between system effectiveness measured by MAP and the precision-based task. A significant, but weak relationship is present for the precision at one document returned metric. A weak relationship is present between MAP and the simple recall-based task.