Multimodal genre classification of TV programs and YouTube videos

• Authors: Hazım Kemal Ekenel;Tomas Semela
• Venue: Multimedia Tools and Applications
• Year: 2013

This paper presents an automatic video genre classification system, which utilizes several low level audio-visual features as well as cognitive and structural information, and in case of web videos tag-based features, to classify the types of TV programs and YouTube videos. Classification is performed using an ensemble of support vector machines. The visual descriptors consist of color and texture-based features, which are often used to represent the concepts appearing in a video. The audio descriptors are signal energy, zero crossing rate, fundamental frequency, and mel-frequency cepstral coefficients representing a wide range of perceptual cues available in the audio signal. Cognitive descriptors correspond to the information derived from a face detector, whereas structural descriptors are related to shot editing of the video. Tag descriptor is used additionally for the genre classification of YouTube videos and it is based on term frequency-inverse document frequency measure. For each feature and type of genre a separate support vector machine classifier is trained following the one-vs-all scheme. The outputs of the classifiers are then combined to yield the final classification result. The proposed system is extensively evaluated using complete TV programs from Italian RAI TV channel, from a French TV channel, and videos from YouTube on which using only the audio-visual cues as well as cognitive and structural information, 99.2, 94.5 and 87.3% correct classification rates are attained, respectively. These results show that the developed system can reliably determine TV programs' genre. Incorporating tag feature to the content-based features increases the YouTube genre classification performance from 87.3 to 89.7%. Further experiments indicate that the quality of videos does not influence the results significantly. It is found that the performance drop in classifying genres of YouTube videos is mainly due to the large variety of content contained in these videos. In summary, this study shows that the proposed low level visual feature set, which we have used to represent the concepts appearing in a video, also provides robust cues for genre classification. In addition, obtained genre information is expected to provide additional cues which can be used to improve the concept detection system's performance. It has also been shown that ensemble of support vector machine classifiers outperforms neural network based classification proposed in the previous state-of-the-art genre classification systems (Montagnuolo and Messina, AIIA, LNAI 4733:730---741, 2007, Multimed Tools Appl 41(1):125---159, 2009). Besides the improvement in the employed feature set and classification scheme, the experimental framework of the study is exemplary with the extensive tests conducted on different domains ranging from TV programs from different countries to web videos.