Explaining User Performance in Information Retrieval: Challenges to IR Evaluation

Authors:
Kalervo Järvelin
Affiliations:
University of Tampere, Finland
Venue:
ICTIR '09 Proceedings of the 2nd International Conference on Theory of Information Retrieval: Advances in Information Retrieval Theory
Year:
2009

Citing 9
Cited 8

Relevance and retrieval evaluation: perspectives from medicine

Journal of the American Society for Information Science - Special issue: relevance research
Users lost: reflections on the past, future, and limits of information science

ACM SIGIR Forum
Real life, real users, and real needs: a study and analysis of user queries on the web

Information Processing and Management: an International Journal
When will information retrieval be "good enough"?

Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval
The Turn: Integration of Information Seeking and Retrieval in Context (The Information Retrieval Series)

The Turn: Integration of Information Seeking and Retrieval in Context (The Information Retrieval Series)
User performance versus precision measures for simple search tasks

SIGIR '06 Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval
An analysis of two approaches in information retrieval: From frameworks to study designs

Journal of the American Society for Information Science and Technology
User adaptation: good results from poor systems

Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval
Discounted cumulated gain based evaluation of multiple-query IR sessions

ECIR'08 Proceedings of the IR research, 30th European conference on Advances in information retrieval

Interactive information seeking via selective application of contextual knowledge

Proceedings of the third symposium on Information interaction in context
An information foraging theory based user study of an adaptive user interaction framework for content-based image retrieval

MMM'11 Proceedings of the 17th international conference on Advances in multimedia modeling - Volume Part II
IR research: systems, interaction, evaluation and theories

ECIR'11 Proceedings of the 33rd European conference on Advances in information retrieval
Find it if you can: a game for modeling different types of web search success using interaction data

Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval
IR research: systems, interaction, evaluation and theories

ACM SIGIR Forum
Evaluation of an adaptive search suggestion system

ECIR'2010 Proceedings of the 32nd European conference on Advances in Information Retrieval
Ranking, relevance judgment, and precision of information retrieval on children's queries: Evaluation of Google, Yahoo!, Bing, Yahoo! Kids, and ask Kids

Journal of the American Society for Information Science and Technology
Influence of timeline and named-entity components on user engagement

ECIR'13 Proceedings of the 35th European conference on Advances in Information Retrieval

Quantified Score

Hi-index	0.00

Visualization

Abstract

The paper makes three points of significance for IR research: (1) The Cranfield paradigm of IR evaluation seems to lose power when one looks at human instead of system performance. (2) Searchers using IR systems in real-life use rather short queries, which individually often have poor performance. However, when used in sessions, they may be surprisingly effective. The searcher's strategies have not been sufficiently described and cannot therefore be properly understood, supported nor evaluated. (3) Searchers in real-life seek to optimize the entire information access process, not just result quality. Evaluation of output alone is insufficient to explain searcher behavior.