Finding and Labeling the Subject of a Captioned Depictive Natural Photograph

• Authors: N. C. Rowe
• Venue: IEEE Transactions on Knowledge and Data Engineering
• Year: 2002

We address the problem of finding the subject of a photographic image intended to illustrate some physical object or objects (驴depictive驴) and taken by usual optical means without magnification (驴natural驴). This could help in developing digital image libraries since important image properties like subject size and color of a photograph are not usually mentioned in accompanying captions and can help rank the photograph retrievals for a user. We explore an approach that identifies the 驴visual focus驴 of the image and the 驴depicted concepts驴 in a caption and connects them. The visual focus is determined by using eight domain-independent characteristics of regions in the segmented image, and the caption depiction is identified by a set a rules applied to the parsed and interpreted caption. The visual-focus determination also does combinatorial optimization on sets of regions to find the set that best satisfies focus criteria. Experiments on 100 randomly selected image-caption pairs show significant improvement in precision of retrieval over simpler methods, and, particularly, emphasizes the value of segmentation of the image.