Towards textually describing complex video contents with audio-visual concept classifiers

  • Authors:

  • Chun Chet Tan;Yu-Gang Jiang;Chong-Wah Ngo

  • Affiliations:

  • City University of Hong Kong, Kowloon, Hong Kong;Fudan University, Shanghai, China;City University of Hong Kong, Kowloon, Hong Kong

  • Venue:
  • MM '11 Proceedings of the 19th ACM international conference on Multimedia
  • Year:
  • 2011

Quantified Score

Hi-index 0.01

Visualization

Abstract

Automatically generating compact textual descriptions of complex video contents has wide applications. With the recent advancements in automatic audio-visual content recognition, in this paper we explore the technical feasibility of the challenging issue of precisely recounting video contents. Based on cutting-edge automatic recognition techniques, we start from classifying a variety of visual and audio concepts in video contents. According to the classification results, we apply simple rule-based methods to generate textual descriptions of video contents. Results are evaluated by conducting carefully designed user studies. We find that the state-of-the-art visual and audio concept classification, although far from perfect, is able to provide very useful clues indicating what is happening in the videos. Most users involved in the evaluation confirmed the informativeness of our machine-generated descriptions.