Communications of the ACM
The weighted majority algorithm
Information and Computation
Human computation tasks with global constraints
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Combining human and machine intelligence in large-scale crowdsourcing
Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems - Volume 1
Crowd IQ: aggregating opinions to boost performance
Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems - Volume 1
Network analysis on provenance graphs from a crowdsourcing application
IPAW'12 Proceedings of the 4th international conference on Provenance and Annotation of Data and Processes
Community-based bayesian aggregation models for crowdsourcing
Proceedings of the 23rd international conference on World wide web
| Hi-index | 0.00 |
In this paper we address the problem of budget allocation for redundantly crowdsourcing a set of classification tasks where a key challenge is to find a trade-off between the total cost and the accuracy of estimation. We propose CrowdBudget, an agent-based budget allocation algorithm, that efficiently divides a given budget among different tasks in order to achieve low estimation error. In particular, we prove that CrowdBudget can achieve at most max{0, K/2- O,(√B)} estimation error with high probability, where K is the number of tasks and B is the budget size. This result significantly outperforms the current best theoretical guarantee from Karger et al,. In addition, we demonstrate that our algorithm outperforms existing methods by up to 40% in experiments based on real-world data from a prominent database of crowdsourced classification responses.