Semantics-preserving bag-of-words models for efficient image annotation

  • Authors:

  • Lei Wu;Steven C.H. Hoi;Nenghai Yu

  • Affiliations:

  • University of Science and Technology of China, Hefei, China;Nanyang Technological University, Singapore, Singapore;University of Science and Technology of China, Hefei, China

  • Venue:
  • LS-MMRM '09 Proceedings of the First ACM workshop on Large-scale multimedia retrieval and mining
  • Year:
  • 2009

Quantified Score

Hi-index 0.00

Visualization

Abstract

The Bag-of-Words (BoW) model is a promising image representation for annotation. One critical limitation of existing BoW models is the semantic loss during the codebook generation process, in which BoW simply clusters visual words in Euclidian space. However, distance between two visual words in Euclidean space does not necessarily reflect the semantic distance between the two concepts, due to the semantic gap between low-level features and high-level semantics. In this paper, we propose a novel scheme for learning a codebook such that semantically related features will be mapped to the same visual word. In particular, we consider the distance between semantically identical features as a measurement of the semantic gap, and attempt to learn an optimized codebook by minimizing this gap. We refer to such a new codebook method as Semantics-Preserving Codebook (SPC) and the corresponding model as Semantics-Preserving Bag-of-Words model (SPBoW). This novel model generates codebook for each object category and only needs to update the codebook for a specific category when incomes an object, which makes it convenient to scale up with the increasing number of objects. Experiments on image annotation tasks with a public testbed from MIT's Labelme project, which contains 11,281 objects of 495 categories, show that the SPC learning scheme is efficient in handling large number of objects and is able to greatly improve the performance of the existing BoW model.