Automatic music video summarization based on audio-visual-text analysis and alignment

  • Authors:

  • Changsheng Xu;Xi Shao;Namunu C. Maddage;Mohan S. Kankanhalli

  • Affiliations:

  • Institute for Infocomm Research, Singapore;Institute for Infocomm Research, Singapore and National University of Singapore, Singapore;Institute for Infocomm Research, Singapore and National University of Singapore, Singapore;National University of Singapore, Singapore

  • Venue:
  • Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval
  • Year:
  • 2005

Quantified Score

Hi-index 0.00

Visualization

Abstract

In this paper, we propose a novel approach for automatic music video summarization based on audio-visual-text analysis and alignment. The music video is separated into the music and video tracks. For the music track, the chorus is detected based on music structure analysis. For the video track, we first segment the shots and classify the shots into close-up face shots and non-face shots, then we extract the lyrics and detect the most repeated lyrics from the shots. The music video summary is generated based on the alignment of boundaries of the detected chorus, shot class and the most repeated lyrics from the music video. The experiments on chorus detection, shot classification, and lyrics detection using 20 English music videos are described. Subjective user studies have been conducted to evaluate the quality and effectiveness of summary. The comparisons with the summaries based on our previous method and the manual method indicate that the results of summarization using the proposed method are better at meeting users' expectations.