Combining image captions and visual analysis for image concept classification

  • Authors:

  • Tomas Kliegr;Krishna Chandramouli;Jan Nemrava;Vojtech Svatek;Ebroul Izquierdo

  • Affiliations:

  • University of Economics, Prague;Queen Mary University, London, United Kingdom;University of Economics, Prague;University of Economics, Prague;Queen Mary University, London, United Kingdom

  • Venue:
  • Proceedings of the 9th International Workshop on Multimedia Data Mining: held in conjunction with the ACM SIGKDD 2008
  • Year:
  • 2008

Quantified Score

Hi-index 0.00

Visualization

Abstract

We present a framework for efficiently exploiting free-text annotations as a complementary resource to image classification. A novel approach called Semantic Concept Mapping (SCM) is used to classify entities occurring in the text to a custom-defined set of concepts. SCM performs unsupervised classification by exploiting the relations between common entities codified in the Wordnet thesaurus. SCM exploits Targeted Hypernym Discovery (THD) to map unknown entities extracted from the text to concepts in Wordnet. We show how the result of SCM/THD can be fused with the outcome of Knowledge Assisted Image Analysis (KAA), a classification algorithm that extracts and labels multiple segments from an image. In the experimental evaluation, THD achieved an accuracy of 75%, and SCM an accuracy of 52%. In one of the first experiments with fusing the results of a free-text and image-content classifier, SCM/THD + KAA achieved a relative improvement of 49% and 31% over the text-only and image-content-only baselines.