User-defined relevance criteria: an exploratory study
Journal of the American Society for Information Science - Special issue: relevance research
Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval
Ant World (demonstration abstract)
Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval
The TREC interactive track: an annotated bibliography
Information Processing and Management: an International Journal - Special issue on interactivity at the text retrieval conference (TREC)
The TREC-5 Confusion Track: Comparing Retrieval Methods for Scanned Text
Information Retrieval
A model for quantitative evaluation of an end-to-end question-answering system
Journal of the American Society for Information Science and Technology
User adaptation: good results from poor systems
Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval
Questionnaires for eliciting evaluation data from users of interactive question answering systems
Natural Language Engineering
Methods for Evaluating Interactive Information Retrieval Systems with Users
Foundations and Trends in Information Retrieval
User-centered evaluation of interactive question answering systems
IQA '06 Proceedings of the Interactive Question Answering Workshop at HLT-NAACL 2006
DESRIST'12 Proceedings of the 7th international conference on Design Science Research in Information Systems: advances in theory and practice
A comprehensive framework for evaluation in design science research
DESRIST'12 Proceedings of the 7th international conference on Design Science Research in Information Systems: advances in theory and practice
Relevance: An improved framework for explicating the notion
Journal of the American Society for Information Science and Technology
Evaluating books finding tools on social media: A case study of aNobii
Information Processing and Management: an International Journal
| Hi-index | 0.00 |
In this article, we introduce a new information system evaluation method and report on its application to a collaborative information seeking system, AntWorld. The key innovation of the new method is to use precisely the same group of users who work with the system as judges, a system we call Cross-Evaluation. In the new method, we also propose to assess the system at the level of task completion. The obvious potential limitation of this method is that individuals may be inclined to think more highly of the materials that they themselves have found and are almost certain to think more highly of their own work product than they do of the products built by others. The keys to neutralizing this problem are careful design and a corresponding analytical model based on analysis of variance. We model the several measures of task completion with a linear model of five effects, describing the users who interact with the system, the system used to finish the task, the task itself, the behavior of individuals as judges, and the self-judgment bias. Our analytical method successfully isolates the effect of each variable. This approach provides a successful model to make concrete the “three-realities” paradigm, which calls for “real tasks,” “real users,” and “real systems.” © 2006 Wiley Periodicals, Inc.