Warping time for more effective real-time crowdsourcing

Authors:
Walter S. Lasecki;Christopher D. Miller;Jeffrey P. Bigham
Affiliations:
University of Rochester, Rochester, New York, USA;University of Rochester, Rochester, New York, USA;University of Rochester, Rochester, New York, USA
Venue:
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Year:
2013

Citing 8
Cited 3

VizWiz: nearly real-time answers to visual questions

UIST '10 Proceedings of the 23nd annual ACM symposium on User interface software and technology
Supporting collaborative transcription of recorded speech with a 3D game interface

KES'10 Proceedings of the 14th international conference on Knowledge-based and intelligent information and engineering systems: Part IV
An overlap-add technique based on waveform similarity (WSOLA) for high quality time-scale modification of speech

ICASSP'93 Proceedings of the 1993 IEEE international conference on Acoustics, speech, and signal processing: speech processing - Volume II
Real-time crowd control of existing interfaces

Proceedings of the 24th annual ACM symposium on User interface software and technology
Crowds in two seconds: enabling realtime crowd-powered interfaces

Proceedings of the 24th annual ACM symposium on User interface software and technology
Real-time captioning by groups of non-experts

Proceedings of the 25th annual ACM symposium on User interface software and technology
Online quality control for real-time crowd captioning

Proceedings of the 14th international ACM SIGACCESS conference on Computers and accessibility
Real-time crowd labeling for deployable activity recognition

Proceedings of the 2013 conference on Computer supported cooperative work

Legion scribe: real-time captioning by the non-experts

Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility
Real-time captioning by non-experts with legion scribe

Proceedings of the 15th International ACM SIGACCESS Conference on Computers and Accessibility
Accessibility Evaluation of Classroom Captions

ACM Transactions on Accessible Computing (TACCESS)

Quantified Score

Hi-index	0.01

Visualization

Abstract

In this paper, we introduce the idea of "warping time" to improve crowd performance on the difficult task of captioning speech in real-time. Prior work has shown that the crowd can collectively caption speech in real-time by merging the partial results of multiple workers. Because non-expert workers cannot keep up with natural speaking rates, the task is frustrating and prone to errors as workers buffer what they hear to type later. The TimeWarp approach automatically increases and decreases the speed of speech playback systematically across individual workers who caption only the periods played at reduced speed. Studies with 139 remote crowd workers and 24 local participants show that this approach improves median coverage (14.8%), precision (11.2%), and per-word latency (19.1%). Warping time may also help crowds outperform individuals on other difficult real-time performance tasks.