Using Multi-Modal Semantic Association Rules to fuse keywords and visual features automatically for Web image retrieval

  • Authors:

  • Ruhan He;Naixue Xiong;Laurence T. Yang;Jong Hyuk Park

  • Affiliations:

  • College of Computer Science, Wuhan University of Science and Engineering, Wuhan 430073, China;Department of Computer Science, Georgia State University, Atlanta, GA 30303, USA;Department of Computer Science, St. Francis Xavier University, Antigonish, Canada;Department of Computer Science and Engineering, Seoul National University of Technology, 172 Gongreung 2-dong, Nowon-gu, Seoul 139-742, Republic of Korea

  • Venue:
  • Information Fusion
  • Year:
  • 2011

Quantified Score

Hi-index 0.00

Visualization

Abstract

A recent trend for image search is to fuse the two basic modalities of Web images, i.e., textual features (usually represented by keywords) and visual features for retrieval. The key issue is how to associate the two modalities for fusion. In this paper, a new approach based on Multi-Modal Semantic Association Rule (MMSAR) is proposed to fuse keywords and visual features automatically for Web image retrieval. A MMSAR contains a single keyword and several visual feature clusters, which crosses and associates the two modalities of Web images. A customized frequent itemsets mining algorithm is designed for the particular MMSARs based on the existing inverted file, and a new support-confidence framework is defined for the mining algorithm. Based on the mined MMSARs, the keywords and the visual features are fused automatically in the retrieval process. The proposed approach not only remarkably improves the retrieval precision, but also has fast response time. The experiments are carried out in a Web image retrieval system, VAST (VisuAl & SemanTic image search), and the results show the superiority and effectiveness of the proposed approach.