Harnessing the wisdom of crowds in wikipedia: quality through coordination
Proceedings of the 2008 ACM conference on Computer supported cooperative work
Financial incentives and the "performance of crowds"
Proceedings of the ACM SIGKDD Workshop on Human Computation
TurKit: human computation algorithms on mechanical turk
UIST '10 Proceedings of the 23nd annual ACM symposium on User interface software and technology
Soylent: a word processor with a crowd inside
UIST '10 Proceedings of the 23nd annual ACM symposium on User interface software and technology
VizWiz: nearly real-time answers to visual questions
UIST '10 Proceedings of the 23nd annual ACM symposium on User interface software and technology
Human computation: a survey and taxonomy of a growing field
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Crowdsourcing translation: professional quality from non-professionals
HLT '11 Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1
Real-time crowd control of existing interfaces
Proceedings of the 24th annual ACM symposium on User interface software and technology
Crowds in two seconds: enabling realtime crowd-powered interfaces
Proceedings of the 24th annual ACM symposium on User interface software and technology
Real-time captioning by groups of non-experts
Proceedings of the 25th annual ACM symposium on User interface software and technology
Pairwise ranking aggregation in a crowdsourced setting
Proceedings of the sixth ACM international conference on Web search and data mining
Real-time crowd labeling for deployable activity recognition
Proceedings of the 2013 conference on Computer supported cooperative work
EmailValet: managing email overload through private, accountable crowdsourcing
Proceedings of the 2013 conference on Computer supported cooperative work
Proceedings of the 2013 conference on Computer supported cooperative work
Chorus: a crowd-powered conversational assistant
Proceedings of the 26th annual ACM symposium on User interface software and technology
Fine-Grained Crowdsourcing for Fine-Grained Recognition
CVPR '13 Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition
Answering visual questions with conversational crowd assistants
Proceedings of the 15th International ACM SIGACCESS Conference on Computers and Accessibility
| Hi-index | 0.00 |
Crowd-powered systems have become a popular way to augment the capabilities of automated systems in real-world settings. Many of these systems rely on human workers to process potentially sensitive data or make important decisions. This puts these systems at risk of unintentionally releasing sensitive data or having their outcomes maliciously manipulated. While almost all crowd-powered approaches account for errors made by individual workers, few factor in active attacks on the system. In this paper, we analyze different forms of threats from individuals and groups of workers extracting information from crowd-powered systems or manipulating these systems' outcomes. Via a set of studies performed on Amazon's Mechanical Turk platform and involving 1,140 unique workers, we demonstrate the viability of these threats. We show that the current system is vulnerable to coordinated attacks on a task based on the requests of another task and that a significant portion of Mechanical Turk workers are willing to contribute to an attack. We propose several possible approaches to mitigating these threats, including leveraging workers who are willing to go above and beyond to help, automatically flagging sensitive content, and using workflows that conceal information from each individual, while still allowing the group to complete a task. Our findings enable the crowd to continue to play an important part in automated systems, even as the data they use and the decisions they support become increasingly important.