The Nonstochastic Multiarmed Bandit Problem
SIAM Journal on Computing
Finite-Time Regret Bounds for the Multiarmed Bandit Problem
ICML '98 Proceedings of the Fifteenth International Conference on Machine Learning
Approximation Algorithms for Partial-Information Based Stochastic Control with Markovian Rewards
FOCS '07 Proceedings of the 48th Annual IEEE Symposium on Foundations of Computer Science
Analyzing the Amazon Mechanical Turk marketplace
XRDS: Crossroads, The ACM Magazine for Students - Comp-YOU-Ter
Human computation: a survey and taxonomy of a growing field
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Sustainable employment in India by crowdsourcing enterprise tasks
Proceedings of the 3rd ACM Symposium on Computing for Development
CrowdUtility: know the crowd that works for you
CHI '13 Extended Abstracts on Human Factors in Computing Systems
| Hi-index | 0.00 |
Crowdsourcing is rapidly emerging as a computing paradigm that can employ the collective intelligence of a distributed human population to solve a wide variety of tasks. However, unlike organizational environments where workers have set work hours, known skill sets and performance indicators that can be monitored and controlled, most crowdsourcing platforms leverage the capabilities of fleeting workers who exhibit changing work patterns, expertise, and quality of work. Consequently, platforms exhibit significant variability in terms of performance characteristics (like response time, accuracy, and completion rate). While this variability has been folklore in the crowdsourcing community, we are the first to show data that displays this kind of changing behavior. Notably, these changes are not due to a distribution with high variance; rather, the distribution itself is changing over time. Deciding which platform is most suitable given the requirements of a task is of critical importance in order to optimize performance; further, making the decision(s) adaptively to accommodate the dynamically changing crowd characteristics is a problem that has largely been ignored. In this paper, we address the changing crowds problem and, specifically, propose a multi-armed bandit based framework. We introduce the simple epsilon-smart algorithm that performs robustly. Counterfactual results based on real-life data from two popular crowd platforms demonstrate the efficacy of the proposed approach. Further simulations using a random-walk model for crowd performance demonstrate its scalability and adaptability to more general scenarios.