Joint visual-text modeling for automatic retrieval of multimedia documents

  • Authors:

  • G. Iyengar;P. Duygulu;S. Feng;P. Ircing;S. P. Khudanpur;D. Klakow;M. R. Krause;R. Manmatha;H. J. Nock;D. Petkova;B. Pytlik;P. Virga

  • Affiliations:

  • IBM TJ Watson Research Center;Bilkent University;University of Massachusetts, Amherst;Univ. West Bohemia;Johns Hopkins University;Saarland University;Georgetown University;University of Massachusetts, Amherst;IBM TJ Watson Research Center;Mt. Holyoke College;Johns Hopkins University;Johns Hopkins University

  • Venue:
  • Proceedings of the 13th annual ACM international conference on Multimedia
  • Year:
  • 2005

Quantified Score

Hi-index 0.00

Visualization

Abstract

In this paper we describe a novel approach for jointly modeling the text and the visual components of multimedia documents for the purpose of information retrieval(IR). We propose a novel framework where individual components are developed to model different relationships between documents and queries and then combined into a joint retrieval framework. In the state-of-the-art systems, a late combination between two independent systems, one analyzing just the text part of such documents, and the other analyzing the visual part without leveraging any knowledge acquired in the text processing, is the norm. Such systems rarely exceed the performance of any single modality (i.e. text or video) in information retrieval tasks. Our experiments indicate that allowing a rich interaction between the modalities results in significant improvement in performance over any single modality. We demonstrate these results using the TRECVID03 corpus, which comprises 120 hours of broadcast news videos. Our results demonstrate over 14 % improvement in IR performance over the best reported text-only baseline and ranks amongst the best results reported on this corpus.