Effects of audio and visual surrogates for making sense of digital video

  • Authors:

  • Yaxiao Song;Gary Marchionini

  • Affiliations:

  • University of North Carolina at Chapel Hill, Chapel Hill, NC;University of North Carolina at Chapel Hill, Chapel Hill, NC

  • Venue:
  • Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
  • Year:
  • 2007

Quantified Score

Hi-index 0.01

Visualization

Abstract

Video surrogates are meant to help people quickly make sense of the content of a video before downloading or seeking more detailed information. In this paper we present the results of a study comparing the effectiveness of three different surrogates for objects in digital video libraries. Thirty-six people participated in a within subjects user study in which they did five tasks for each of three surrogate alternatives: visual alone (a storyboard), audio alone (spoken description), and combined visual and audio (a storyboard augmented with spoken description). The results show that combined surrogates are more effective, strongly preferred, and do not penalize efficiency. The results also demonstrate that spoken descriptions alone lead to better understanding of the video segments than do visual storyboards alone, although people like to have visual surrogates and use them to confirm interpretations and add context. Participants were able to easily use the combined surrogates even though they were not synchronized, suggesting that synchronization of different media channels may not be necessary in surrogates as it is in full video. The results suggest that multimodal surrogates should be incorporated into video retrieval user interfaces and audio surrogates should be used in small display interfaces. The study also raises questions about the need to synchronize different information channels in multimedia surrogates.