Online quality control for real-time crowd captioning

  • Authors:

  • Walter S. Lasecki;Jeffrey P. Bigham

  • Affiliations:

  • University of Rochester, Rochester, NY, USA;University of Rochester, Rochester, NY, USA

  • Venue:
  • Proceedings of the 14th international ACM SIGACCESS conference on Computers and accessibility
  • Year:
  • 2012

Quantified Score

Hi-index 0.00

Visualization

Abstract

Approaches for real-time captioning of speech are either expensive (professional stenographers) or error-prone (automatic speech recognition). As an alternative approach, we have been exploring whether groups of non-experts can collectively caption speech in real-time. In this approach, each worker types as much as they can and the partial captions are merged together in real-time automatically. This approach works best when partial captions are correct and received within a few seconds of when they were spoken, but these assumptions break down when engaging workers on-demand from existing sources of crowd work like Amazon's Mechanical Turk. In this paper, we present methods for quickly identifying workers who are producing good partial captions and estimating the quality of their input. We evaluate these methods in experiments run on Mechanical Turk in which a total of 42 workers captioned 20 minutes of audio. The methods introduced in this paper were able to raise overall accuracy from 57.8% to 81.22% while keeping coverage of the ground truth signal nearly unchanged.