Towards correct and informative evaluation methodology for texture classification under varying viewpoint and illumination

  • Authors:

  • Ondřej Drbohlav;Aleš Leonardis

  • Affiliations:

  • Czech Technical University in Prague, Faculty of Electrical Engineering, Department of Cybernetics, Technická 2, 166 27 Praha 6, Czech Republic;University of Ljubljana, Faculty of Computer and Information Science, Visual Cognitive Systems Laboratory, Traška 25, SI-1001 Ljubljana, Slovenia

  • Venue:
  • Computer Vision and Image Understanding
  • Year:
  • 2010

Quantified Score

Hi-index 0.01

Visualization

Abstract

3D texture classification under varying viewpoint and illumination has been a vivid research topic, and many methods have been developed. It is crucial that these methods be compared using an unbiased evaluation methodology. The most frequently employed methodologies use images from the Columbia-Utrecht Reflectance and Texture Database. These methodologies construct the training and test sets to be disjoint in the imaging parameters, but do not separate them spatially because they use images of the same surface patch for both. We perform a series of experiments which show that such practice leads to overestimation of classifier performance and distorts experimental findings. To correct that, we accurately register the images across all imaging conditions and split the surface patches to parts. The training and testing is then done on spatially disjoint parts. We show that such methodology gives a more realistic assessment of classifier performance. The sample annotations for all images are publicly available.