Usage analysis of a digital library
Proceedings of the third ACM conference on Digital libraries
Engineering a multi-purpose test collection for web retrieval experiments
Information Processing and Management: an International Journal
Accurately interpreting clickthrough data as implicit feedback
Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval
MonetDB/XQuery: a fast XQuery processor powered by a relational engine
Proceedings of the 2006 ACM SIGMOD international conference on Management of data
Investigating the querying and browsing behavior of advanced search engine users
SIGIR '07 Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval
Evaluation of digital libraries
International Journal on Digital Libraries
On the history of evaluation in IR
Journal of Information Science
Focused Search in Digital Archives
WISE '09 Proceedings of the 10th International Conference on Web Information Systems Engineering
A model to estimate intrinsic document relevance from the clickthrough logs of a web search engine
Proceedings of the third ACM international conference on Web search and data mining
Searching archival finding aids: retrieval in original order?
ECDL'09 Proceedings of the 13th European conference on Research and advanced technology for digital libraries
Search log analysis of user stereotypes, information seeking behavior, and contextual evaluation
Proceedings of the third symposium on Information interaction in context
AutoEval: an evaluation methodology for evaluating query suggestions using query logs
ECIR'11 Proceedings of the 33rd European conference on Advances in information retrieval
Estimating interleaved comparison outcomes from historical click data
Proceedings of the 21st ACM international conference on Information and knowledge management
Quantitative analysis of search sessions enhanced by gaze tracking with dynamic areas of interest
TPDL'12 Proceedings of the Second international conference on Theory and Practice of Digital Libraries
Fidelity, Soundness, and Efficiency of Interleaved Comparison Methods
ACM Transactions on Information Systems (TOIS)
Hi-index | 0.00 |
Anyone offering content in a digital library is naturally interested in assessing its performance: how well does my system meet the users' information needs? Standard evaluation benchmarks have been developed in information retrieval that can be used to test retrieval effectiveness. However, these generic benchmarks focus on a single document genre, language, media-type, and searcher stereotype that is radically different from the unique content and user community of a particular digital library. This paper proposes to derive a domain-specific test collection from readily available interaction data in search logs files that captures the domain-specificity of digital libraries. We use as case study an archival institution's complete search log that spans over multiple years, and derive a large-scale test collection. We manually derive a set of topics judged by human experts--based on a set of e-mail reference questions and responses from archivists--and use this for validation. Our main finding is that we can derive a reliable and domain-specific test collection from search log files.