A Test Collection for the Evaluation of Content-Based Image Retrieval Algorithms—A User and Task-Based Approach

  • Authors:

  • Marjo Markkula;Marius Tico;Bemmu Sepponen;Katja Nirkkonen;Eero Sormunen

  • Affiliations:

  • Department of Information Studies, University of Tampere, Finland. marjo.markkula@uta.fi;Digital Media Institute, Tampere University of Technology, Finland. tico@cs.tut.fi;Department of Information Services, University of Tampere, Finland;Department of Information Services, University of Tampere, Finland;Department of Information Studies, University of Tampere, Finland. eero.sormunen@uta.fi

  • Venue:
  • Information Retrieval
  • Year:
  • 2001

Quantified Score

Hi-index 0.00

Visualization

Abstract

Content-based image retrieval (CBIR) algorithms have been seen as a promising access method for digital photograph collections. Unfortunately, we have very little evidence of the usefulness of these algorithms in real user needs and contexts. In this paper, we introduce a test collection for the evaluation of CBIR algorithms. In the test collection, the performance testing is based on photograph similarity perceived by end-users in the context of realistic illustration tasks and environment. The building process and the characteristics of the resulting test collection are outlined, including a typology of similarity criteria expressed by the subjects judging the similarity of photographs. A small-scale study on the consistency of similarity assessments is presented. A case evaluation of two CBIR algorithms is reported. The results show clear correlation between the subjects' similarity assessments and the functioning of feature parameters of the tested algorithms.