Evaluation measures for interactive information retrieval
Information Processing and Management: an International Journal - Special issue on evaluation issues in information retrieval
Perceptual speed, learning and information retrieval performance
SIGIR '94 Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval
Evaluating interactive systems in TREC
Journal of the American Society for Information Science - Special issue: evaluation of information retrieval systems
Time, relevance and interaction modelling for information retrieval
Proceedings of the 20th annual international ACM SIGIR conference on Research and development in information retrieval
Variations in relevance judgments and the measurement of retrieval effectiveness
Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval
Overview of the sixth text REtrieval conference (TREC-6)
Information Processing and Management: an International Journal - The sixth text REtrieval conference (TREC-6)
Why batch and user evaluations do not give the same results
Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval
Some(what) grand challenges for information retrieval
ACM SIGIR Forum
Evaluating web search using task completion time
Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval
Methods for Evaluating Interactive Information Retrieval Systems with Users
Foundations and Trends in Information Retrieval
Search User Interfaces
ACM Transactions on Information Systems (TOIS)
A review of factors influencing user satisfaction in information retrieval
Journal of the American Society for Information Science and Technology
| Hi-index | 0.00 |
User satisfaction, though difficult to measure, is the main goal of Information Retrieval (IR) systems. In recent years, as Interactive Information Retrieval (IIR) systems have become increasingly popular, user effectiveness also has become critical in evaluating IIR systems. However, existing measures in IR evaluation are not particularly suitable for gauging user satisfaction and user effectiveness. In this paper, we propose two new measures to evaluate IIR systems, the Normalized Task Completion Time (NT) and the Normalized User Effectiveness (NUE). The two measures overcome limitations of existing measures and are efficient to calculate in that they do not need a large pool of search tasks. A user study was conducted to investigate the relationships between the two measures and the user satisfaction and effectiveness of a given IR system. The learning effects described by NT, NUE, and the task completion time were also studied and compared. The results show that NT is strongly correlated with user satisfaction, NUE is a better indicator of system effectiveness than task completion time, and both new measures are superior to task completion time in describing the learning effect of the given IR system.