Workflow Automation: Overview and Research Issues
Information Systems Frontiers
Computer
Get another label? improving data quality and data mining using multiple, noisy labelers
Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining
Toward automatic task design: a progress report
Proceedings of the ACM SIGKDD Workshop on Human Computation
TurKit: human computation algorithms on mechanical turk
UIST '10 Proceedings of the 23nd annual ACM symposium on User interface software and technology
Soylent: a word processor with a crowd inside
UIST '10 Proceedings of the 23nd annual ACM symposium on User interface software and technology
VizWiz: nearly real-time answers to visual questions
UIST '10 Proceedings of the 23nd annual ACM symposium on User interface software and technology
Analyzing the Amazon Mechanical Turk marketplace
XRDS: Crossroads, The ACM Magazine for Students - Comp-YOU-Ter
Human computation: a survey and taxonomy of a growing field
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
CrowdDB: answering queries with crowdsourcing
Proceedings of the 2011 ACM SIGMOD International Conference on Management of data
Real-time crowd control of existing interfaces
Proceedings of the 24th annual ACM symposium on User interface software and technology
CrowdForge: crowdsourcing complex work
Proceedings of the 24th annual ACM symposium on User interface software and technology
The jabberwocky programming environment for structured social computing
Proceedings of the 24th annual ACM symposium on User interface software and technology
CrowdWeaver: visually managing complex crowd work
Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work
Strategies for crowdsourcing social data analysis
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Navigating by index and guided tour for fact finding
Proceedings of the 30th ACM international conference on Design of communication
CrowdScape: interactively visualizing user behavior and output
Proceedings of the 25th annual ACM symposium on User interface software and technology
Workflow transparency in a microtask marketplace
Proceedings of the 17th ACM international conference on Supporting group work
Enhancing reliability using peer consistency evaluation in human computation
Proceedings of the 2013 conference on Computer supported cooperative work
Crowd vs. crowd: large-scale cooperative design through open team competition
Proceedings of the 2013 conference on Computer supported cooperative work
Co-worker transparency in a microtask marketplace
Proceedings of the 2013 conference on Computer supported cooperative work
Cascade: crowdsourcing taxonomy creation
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Multiverse: crowd algorithms on existing interfaces
CHI '13 Extended Abstracts on Human Factors in Computing Systems
CrowdStudy: general toolkit for crowdsourced evaluation of web interfaces
Proceedings of the 5th ACM SIGCHI symposium on Engineering interactive computing systems
Collaborative problem solving: a study of MathOverflow
Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing
Gamifying citizen science: a study of two user groups
Proceedings of the companion publication of the 17th ACM conference on Computer supported cooperative work & social computing
Sharing Knowledge and Expertise: The CSCW View of Knowledge Management
Computer Supported Cooperative Work
Hi-index | 0.00 |
Preparing complex jobs for crowdsourcing marketplaces requires careful attention to workflow design, the process of decomposing jobs into multiple tasks, which are solved by multiple workers. Can the crowd help design such workflows? This paper presents Turkomatic, a tool that recruits crowd workers to aid requesters in planning and solving complex jobs. While workers decompose and solve tasks, requesters can view the status of worker-designed workflows in real time; intervene to change tasks and solutions; and request new solutions to subtasks from the crowd. These features lower the threshold for crowd employers to request complex work. During two evaluations, we found that allowing the crowd to plan without requester supervision is partially successful, but that requester intervention during workflow planning and execution improves quality substantially. We argue that Turkomatic's collaborative approach can be more successful than the conventional workflow design process and discuss implications for the design of collaborative crowd planning systems.