Context-based multi-label image annotation

• Authors: Zhiwu Lu;Horace H. S. Ip;Qizhen He
• Venue: Proceedings of the ACM International Conference on Image and Video Retrieval
• Year: 2009

This paper presents a novel context-based keyword propagation method for automatic image annotation. We follow the idea of keyword propagation and formulate image annotation as a multi-label learning problem, which is further resolved efficiently by linear programming. In this way, our method can exploit the context between keywords during keyword propagation. Unlike the popular relevance models that treat each keyword independently, our method can simultaneously propagate multiple keywords (i.e. labels) from the training images to the test images using their similarities. Moreover, we present a new 2D string kernel, called spatial spectrum kernel, to take into account another type of context when defining the similarity between images for keyword propagation. Each image is first denoted as a 2D sequence of visual keywords which are obtained through dividing images into blocks and then clustering these blocks, and a spatial spectrum kernel is then proposed to measure the 2D sequence similarity based on shared occurrences of s-length 1D subsequences through decomposing each 2D sequence into two parallel 1D sequences (i.e. the row-wise and column-wise ones). That is, we incorporate the context between visual keywords into the similarity between images (i.e. 2D sequences) used for keyword propagation. Experiments on two standard image databases demonstrate that the proposed method for automatic image annotation outperforms the state-of-the-art methods.