WordNet: a lexical database for English
Communications of the ACM
Cumulated gain-based evaluation of IR techniques
ACM Transactions on Information Systems (TOIS)
SIMPLIcity: Semantics-sensitive Integrated Matching for Picture Libraries
VISUAL '00 Proceedings of the 4th International Conference on Advances in Visual Information Systems
CVPRW '04 Proceedings of the 2004 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW'04) Volume 12 - Volume 12
About face 3: the essentials of interaction design
About face 3: the essentials of interaction design
On test collections for adaptive information retrieval
Information Processing and Management: an International Journal
Lire: lucene image retrieval: an extensible java CBIR library
MM '08 Proceedings of the 16th ACM international conference on Multimedia
Methods for Evaluating Interactive Information Retrieval Systems with Users
Foundations and Trends in Information Retrieval
Simulating simple and fallible relevance feedback
ECIR'11 Proceedings of the 33rd European conference on Advances in information retrieval
System effectiveness, user models, and user utility: a conceptual framework for investigation
Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval
Managing Event Information: Modeling, Retrieval, and Applications
Managing Event Information: Modeling, Retrieval, and Applications
| Hi-index | 0.00 |
In this paper, we present a document collection with graded relevance assessments that has been sampled from real photographers. In order to reflect both vagueness in the commonly used retrieval calculations as well as in the user's query, we argue that current document collections being based on binary relevance judgments have drawbacks -- in particular if user-centered or relevance feedback-related experiments are conducted. In addition, such system-centric collections are based on documents, which do not necessarily reflect a layperson's personal photo collection. To overcome this issue, we suggest a test set of documents that is based on a study of 19 real photographers discriminating it from Flickr downloads or the like. The collection has been categorized on basis of different criteria such as document quality or motif quality plus the aforementioned graded relevance assessments. Reflecting the photograph taking behavior of the investigated photographers, we are also providing an event-based ground-truth in addition to a topic-based one. In total, 130 different topics are available for the collection. In order to provide means to address different photographer and user types, e.g. in user simulations or in usability engineering, we make the demographic information of both photographers and assessors available. Eventually, this links interactive information retrieval evaluation with persona-based interaction design -- a factor that has been neglected in multimedia information retrieval so far.