Internet-scale collection of human-reviewed data
Proceedings of the 16th international conference on World Wide Web
Unifying Human and Software Services in Web-Scale Collaborations
IEEE Internet Computing
Crowdsourcing and all-pay auctions
Proceedings of the 10th ACM conference on Electronic commerce
SERVICES '09 Proceedings of the 2009 Congress on Services - I
Evolution of two-sided markets
Proceedings of the third ACM international conference on Web search and data mining
Who are the crowdworkers?: shifting demographics in mechanical turk
CHI '10 Extended Abstracts on Human Factors in Computing Systems
Task search in a human computation market
Proceedings of the ACM SIGKDD Workshop on Human Computation
Quality management on Amazon Mechanical Turk
Proceedings of the ACM SIGKDD Workshop on Human Computation
Runtime Behavior Monitoring and Self-Adaptation in Service-Oriented Systems
SASO '10 Proceedings of the 2010 Fourth IEEE International Conference on Self-Adaptive and Self-Organizing Systems
Crowdsourcing systems on the World-Wide Web
Communications of the ACM
Stimulating skill evolution in market-based crowdsourcing
BPM'11 Proceedings of the 9th international conference on Business process management
Optimal crowdsourcing contests
Proceedings of the twenty-third annual ACM-SIAM symposium on Discrete Algorithms
Workflow-Based coordination and cooperation in a service world
ODBASE'06/OTM'06 Proceedings of the 2006 Confederated international conference on On the Move to Meaningful Internet Systems: CoopIS, DOA, GADA, and ODBASE - Volume Part I
Hi-index | 0.00 |
Crowdsourcing is a promising approach for enterprises to maintain a flexible workforce that is able to solve parts of business processes formerly processed in-house. Companies perceive crowdsourcing as a concept that allows receiving solutions quickly and at little cost. Similar to cloud computing where computing power is provided on demand, the crowd promises a flexible on-demand workforce. However, businesses realize that these benefits entail a lack of quality control. The main difference compared to traditional approaches in business process execution is that tasks or activities cannot be directly assigned to employees but are posted to the crowdsourcing platform. Its members can choose deliberately which tasks to book and work on. In fact, crowdsourcing is heavily affected by the loose-coupling of workers to crowdsourcers and the dynamics of the environment. Hence, it remains a major challenge to guarantee high-quality processing of tasks within the prescribed time limit. A further obstacle for adoption of crowdsourcing in enterprises is the fact that it is hard to specify a fair monetary reward in advance. The concepts introduced in this work allow to smoothly integrate new workers, to keep them motivated, and to help them develop and improve skills needed in the system. We present a crowdsourcing marketplace that matches complex tasks, requiring multiple skills, to suitable workers. The key to ensuring high quality lies in skilled members whose capabilities can be estimated correctly. To that end, we present auction mechanisms that help to correctly estimate workers and to evolve skills that are needed in the system. Crowdsourcers do not need to predefine exact prices but only maximum prices they are willing to pay since the actual rewards for tasks are formed by supply and demand. Extensive experiments show that our approach leads to improved crowdsourcing, in most cases.