User effect in evaluating personalized information retrieval systems

Authors:
Effie Lai-Chong Law;Tomaž Klobučar;Matic Pipan
Affiliations:
Institut TIK, ETH Zürich, Zürich, Switzerland;Institut Jožef Stefan, Ljubljana, Slovenia;Institut Jožef Stefan, Ljubljana, Slovenia
Venue:
EC-TEL'06 Proceedings of the First European conference on Technology Enhanced Learning: innovative Approaches for Learning and Knowledge Sharing
Year:
2006

Citing 16
Cited 1

Claude Elwood Shannon: collected papers

Claude Elwood Shannon: collected papers
Toward a theory of user-based relevance: a call for a new paradigm of inquiry

Journal of the American Society for Information Science - Special issue: relevance research
On the measurement of inter-linker consistency and retrieval effectiveness in hypertext databases

SIGIR '94 Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval
Relevance: the whole history

Journal of the American Society for Information Science - Special topic issue on the history of documentation and information science: part II
Organizational usability of digital libraries: case study of legal research in civil and criminal courts

Journal of the American Society for Information Science - Special issue on current research in human-computer interaction
Users' criteria for relevance evaluation: a cross-situational comparison

Information Processing and Management: an International Journal
Measures of relative relevance and ranked half-life: performance indicators for interactive IR

Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval
Dimensions of relevance

Information Processing and Management: an International Journal
Use of relevance criteria across stages of document evaluation: on the complementarity of experimental and naturalistic studies

Journal of the American Society for Information Science and Technology
Evaluation by highly relevant documents

Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval
User perspectives on relevance criteria: a comparison among relevant, partially relevant , and not-relevant judgements

Journal of the American Society for Information Science and Technology
The concept of relevance in IR

Journal of the American Society for Information Science and Technology
Retrieval evaluation with incomplete information

Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval
Technology as Experience

Technology as Experience
Seeking and implementing automated assistance during the search process

Information Processing and Management: an International Journal
Relevance criteria identified by health information users during Web searches: Research Articles

Journal of the American Society for Information Science and Technology

A review of factors influencing user satisfaction in information retrieval

Journal of the American Society for Information Science and Technology

Quantified Score

Hi-index	0.00

Visualization

Abstract

Evaluation of personalized information retrieval (IR) systems is challenged by the user effect, which is manifested in terms of users' inconsistency in relevance judgment, ranking and relevance criteria usage. Two empirical studies on evaluating a personalized search engine were performed. Two types of relative measures computed with different mathematical formulae were compared. The ranking similarity and the randomness of relevance criteria usage were estimated. Results show some undesirable personalization effects. Implications for the future development and research of adaptive IR systems are discussed.