Developing a test collection for the evaluation of integrated search

Authors:
Marianne Lykke;Birger Larsen;Haakon Lund;Peter Ingwersen
Affiliations:
Department of Information Interaction and Information Architecture., Royal School of Library and Information Science, Copenhagen S, Denmark;Department of Information Interaction and Information Architecture., Royal School of Library and Information Science, Copenhagen S, Denmark;Department of Information Interaction and Information Architecture., Royal School of Library and Information Science, Copenhagen S, Denmark;Department of Information Interaction and Information Architecture., Royal School of Library and Information Science, Copenhagen S, Denmark
Venue:
ECIR'2010 Proceedings of the 32nd European conference on Advances in Information Retrieval
Year:
2010

Citing 3
Cited 10

Liberal relevance criteria of TREC -: counting on negligible documents?

SIGIR '02 Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval
An analysis of two approaches in information retrieval: From frameworks to study designs

Journal of the American Society for Information Science and Technology
Eliciting better information need descriptions from users of information search systems

Information Processing and Management: an International Journal

A subjective logic formalisation of the principle of polyrepresentation for information needs

Proceedings of the third symposium on Information interaction in context
Assessors' search result satisfaction associated with relevance in a scientific domain

Proceedings of the third symposium on Information interaction in context
Physicists' information tasks: structure, length and retrieval performance

Proceedings of the third symposium on Information interaction in context
Does degree of work task completion influence retrieval performance?

Proceedings of the 73rd ASIS&T Annual Meeting on Navigating Streams in an Information Ecosystem - Volume 47
Sense discrimination for physics retrieval

Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval
A cascaded classification approach to semantic head recognition

EMNLP '11 Proceedings of the Conference on Empirical Methods in Natural Language Processing
Report on TBAS 2012: workshop on task-based and aggregated search

ACM SIGIR Forum
Preliminary experiments using subjective logic for the polyrepresentation of information needs

Proceedings of the 4th Information Interaction in Context Symposium
An exploratory study into perceived task complexity, topic specificity and usefulness for integrated search

Proceedings of the 4th Information Interaction in Context Symposium
Unsupervised feature adaptation for cross-domain NLP with an application to compositionality grading

CICLing'13 Proceedings of the 14th international conference on Computational Linguistics and Intelligent Text Processing - Volume Part I

Quantified Score

Hi-index	0.00

Visualization

Abstract

The poster discusses the characteristics needed in an information retrieval (IR) test collection to facilitate the evaluation of integrated search, i.e. search across a range of different sources but with one search box and one ranked result list, and describes and analyses a new test collection constructed for this purpose. The test collection consists of approx. 18,000 monographic records, 160,000 papers and journal articles in PDF and 275,000 abstracts with a varied set of metadata and vocabularies from the physics domain, 65 topics based on real work tasks and corresponding graded relevance assessments. The test collection may be used for systems- as well as user-oriented evaluation.