Efficient visual content retrieval and mining in videos

  • Authors:
  • Josef Sivic;Andrew Zisserman

  • Affiliations:
  • Robotics Research Group, Department of Engineering Science, University of Oxford;Robotics Research Group, Department of Engineering Science, University of Oxford

  • Venue:
  • PCM'04 Proceedings of the 5th Pacific Rim Conference on Advances in Multimedia Information Processing - Volume Part II
  • Year:
  • 2004

Quantified Score

Hi-index 0.00

Visualization

Abstract

We describe an image representation for objects and scenes consisting of a configuration of viewpoint covariant regions and their descriptors. This representation enables recognition to proceed successfully despite changes in scale, viewpoint, illumination and partial occlusion. Vector quantization of these descriptors then enables efficient matching on the scale of an entire feature film. We show two applications. The first is to efficient object retrieval where the technology of text retrieval, such as inverted file systems, can be employed at run time to return all shots containing the object in a manner, and with a speed, similar to a Google search for text. The object is specified by a user outlining it in an image, and the object is then delineated in the retrieved shots. The second application is to data mining. We obtain the principal objects, characters and scenes in a video by measuring the reoccurrence of these spatial configurations of viewpoint covariant regions. The applications are illustrated on two full length feature films.