A re-examination of relevance: toward a dynamic, situational definition
Information Processing and Management: an International Journal
The pragmatics of information retrieval experimentation, revisited
Information Processing and Management: an International Journal - Special issue on evaluation issues in information retrieval
Access to nonbook materials: the limits of subject indexing for visual and aural languages
Journal of the American Society for Information Science
A task-oriented approach to information retrieval evaluation
Journal of the American Society for Information Science - Special issue: evaluation of information retrieval systems
Communications of the ACM
Journal of the American Society for Information Science - Special topic issue on the history of documentation and information science: part II
Modeling and retrieving images by content
Information Processing and Management: an International Journal
Intelligent image databases: towards advanced image retrieval
Intelligent image databases: towards advanced image retrieval
Attributes of images in describing tasks
Information Processing and Management: an International Journal
Spatial querying for image retrieval: a user-oriented evaluation
Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval
From highly relevant to not relevant: examining different regions of relevance
Information Processing and Management: an International Journal
Information Retrieval Experiment
Information Retrieval Experiment
End-User Searching Challenges Indexing Practices inthe Digital Newspaper Photo Archive
Information Retrieval
Finding Pictures of Objects in Large Collections of Images
ECCV '96 Proceedings of the International Workshop on Object Representation in Computer Vision II
End-User Searching Challenges Indexing Practices inthe Digital Newspaper Photo Archive
Information Retrieval
Creative professional users' musical relevance criteria
Journal of Information Science
Comparison of categorization criteria across image genres
Proceedings of the 73rd ASIS&T Annual Meeting on Navigating Streams in an Information Ecosystem - Volume 47
Development and evaluation of a multifaceted magazine image categorization model
Journal of the American Society for Information Science and Technology
Hi-index | 0.00 |
Content-based image retrieval (CBIR) algorithms have been seen as a promising access method for digital photo collections, sooner or later replacing the traditional text-based methods. Unfortunately, we have very little evidence of the usefulness of these algorithms in real user needs and contexts. One problem is that appropriately designed test collections are not available even for the basic performance testing of the CBIR algorithms. This paper proposes a task-oriented evaluation framework and an efficient procedure for constructing test collections for CBIR algorithms. First, the paper defines a plausible function for these algorithms in general purpose photo retrieval systems. We believe that the CBIR algorithms could be applied effectively in conjunction with text-based photo retrieval. Text-based methods are powerful in retrieving topically related items but do not support browsing. The CBIR algorithms could help in identifying visually similar photos within (often large) result sets of textual queries. The proposed evaluation framework is based on the concept of perceived similarity and emphasises the role of expertise and realistic illustration tasks as a premise of similarity assessments. A major innovation of the proposed test collection is that it consists of an array of small test sets each built up of a tiny database, a query photo, and respective similarity assessments. The approach supports testing of prototype CBIR algorithms in short development cycles. The empirical part of the paper reports how journalists were judging the similarity of photos while searching in the course of simulated, but realistic illustration tasks. The goal of the study was to exercise the construction process of the test collection. The results show that the task-oriented evaluation framework and the proposed procedures for constructing the test collection can be successfully applied. The lessons learned from the simulated illustration tasks, collection of similarity assessments and construction of the test collection are discussed.