A structured learning approach to semantic photo indexing and query

  • Authors:
  • Joo-Hwee Lim;Jesse S. Jin;Suhuai Luo

  • Affiliations:
  • Institute for Infocomm Research, Singapore;University of Newcastle, Callaghan, Australia;University of Newcastle, Callaghan, Australia

  • Venue:
  • AIRS'05 Proceedings of the Second Asia conference on Asia Information Retrieval Technology
  • Year:
  • 2005

Quantified Score

Hi-index 0.00

Visualization

Abstract

Consumer photos exhibit highly varied contents, diverse resolutions and inconsistent quality. The objects are usually ill-posed, occluded, and cluttered with poor lighting, focus, and exposure. Existing image retrieval approaches face many obstacles such as robust object segmentation, small sampling problem during relevance feedback, semantic gap between low-level features and high-level semantics, etc. We propose a structured learning approach to design domain-relevant visual semantics, known as semantic support regions, to support semantic indexing and visual query for consumer photos. Semantic support regions are segmentation-free image regions that exhibit semantic meanings and that can be learned statistically to span a new indexing space. They are detected from image content, reconciled across multiple resolutions, and aggregated spatially to form local semantic histograms. Query by Spatial Icons (QBSI) is a unique visual query language to specify semantic icons and spatial extents in a Boolean expression. Based on 2400 heterogeneous consumer photos and 26 semantic support regions learned from a small training set, we demonstrate the usefulness of the visual query language with 15 QBSI queries that have attained high precision values at top retrieved images.