Execution control for crowdsourcing

Authors:
Daniel S. Weld; Mausam;Peng Dai
Affiliations:
University of Washington, Seattle, WA, USA;University of Washington, Seattle, USA;University of Washington, Seattle, USA
Venue:
Proceedings of the 24th annual ACM symposium adjunct on User interface software and technology
Year:
2011

Citing 5
Cited 0

TurKit: tools for iterative tasks on mechanical Turk

Proceedings of the ACM SIGKDD Workshop on Human Computation
Soylent: a word processor with a crowd inside

UIST '10 Proceedings of the 23nd annual ACM symposium on User interface software and technology
VizWiz: nearly real-time answers to visual questions

UIST '10 Proceedings of the 23nd annual ACM symposium on User interface software and technology
Crowds in two seconds: enabling realtime crowd-powered interfaces

Proceedings of the 24th annual ACM symposium on User interface software and technology
Bandit based monte-carlo planning

ECML'06 Proceedings of the 17th European conference on Machine Learning

Quantified Score

Hi-index	0.00

Visualization

Abstract

Crowdsourcing marketplaces enable a wide range of applications, but constructing any new application is challenging - usually requiring a complex, self-managing workflow in order to guarantee quality results. We report on the CLOWDER project, which uses machine learning to continually refine models of worker performance and task difficulty. We present decision-theoretic optimization techniques that can select the best parameters for a range of workflows. Initial experiments show our optimized workflows are significantly more economical than with manually set parameters.