Shepherding the crowd: managing and providing feedback to crowd workers

  • Authors:

  • Steven Dow;Anand Kulkarni;Brie Bunge;Truc Nguyen;Scott Klemmer;Björn Hartmann

  • Affiliations:

  • Stanford University, Stanford, CA, USA;University of California, Berkeley, Berkeley, CA, USA;Stanford University, Stanford, CA, USA;Stanford University, Stanford, CA, USA;Stanford University, Stanford, CA, USA;University of California, Berkeley, Berkeley, CA, USA

  • Venue:
  • CHI '11 Extended Abstracts on Human Factors in Computing Systems
  • Year:
  • 2011

Quantified Score

Hi-index 0.00

Visualization

Abstract

Micro-task platforms provide a marketplace for hiring people to do short-term work for small payments. Requesters often struggle to obtain high-quality results, especially on content-creation tasks, because work cannot be easily verified and workers can move to other tasks without consequence. Such platforms provide little opportunity for workers to reflect and improve their task performance. Timely and task-specific feedback can help crowd workers learn, persist, and produce better results. We analyze the design space for crowd feedback and introduce Shepherd, a prototype system for visualizing crowd work, providing feedback, and promoting workers into shepherding roles. This paper describes our current progress and our plans for system development and evaluation.