IGOR: a system for program debugging via reversible execution
PADD '88 Proceedings of the 1988 ACM SIGPLAN and SIGOPS workshop on Parallel and distributed debugging
Labeling images with a computer game
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Becoming Wikipedian: transformation of participation in a collaborative online encyclopedia
GROUP '05 Proceedings of the 2005 international ACM SIGGROUP conference on Supporting group work
Computer
LabelMe: A Database and Web-Based Tool for Image Annotation
International Journal of Computer Vision
Crowdsourcing user studies with Mechanical Turk
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Harnessing the wisdom of crowds in wikipedia: quality through coordination
Proceedings of the 2008 ACM conference on Computer supported cooperative work
Finding causes of program output with the Java Whyline
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Financial incentives and the "performance of crowds"
Proceedings of the ACM SIGKDD Workshop on Human Computation
Cheap and fast---but is it good?: evaluating non-expert annotations for natural language tasks
EMNLP '08 Proceedings of the Conference on Empirical Methods in Natural Language Processing
Crowdsourcing graphical perception: using mechanical turk to assess visualization design
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Exploring iterative and parallel human computation processes
Proceedings of the ACM SIGKDD Workshop on Human Computation
Soylent: a word processor with a crowd inside
UIST '10 Proceedings of the 23nd annual ACM symposium on User interface software and technology
Soylent: a word processor with a crowd inside
UIST '10 Proceedings of the 23nd annual ACM symposium on User interface software and technology
VizWiz: nearly real-time answers to visual questions
UIST '10 Proceedings of the 23nd annual ACM symposium on User interface software and technology
XRDS: Crossroads, The ACM Magazine for Students - Comp-YOU-Ter
Audiowiz: nearly real-time audio transcriptions
Proceedings of the 12th international ACM SIGACCESS conference on Computers and accessibility
The trouble with social computing systems research
CHI '11 Extended Abstracts on Human Factors in Computing Systems
A crowdsourcing model for receiving design critique
CHI '11 Extended Abstracts on Human Factors in Computing Systems
Turkomatic: automatic recursive task and workflow design for mechanical turk
CHI '11 Extended Abstracts on Human Factors in Computing Systems
Demonstration of Qurk: a query processor for humanoperators
Proceedings of the 2011 ACM SIGMOD International Conference on Management of data
Platemate: crowdsourcing nutritional analysis from food photographs
Proceedings of the 24th annual ACM symposium on User interface software and technology
Real-time crowd control of existing interfaces
Proceedings of the 24th annual ACM symposium on User interface software and technology
Crowds in two seconds: enabling realtime crowd-powered interfaces
Proceedings of the 24th annual ACM symposium on User interface software and technology
CrowdForge: crowdsourcing complex work
Proceedings of the 24th annual ACM symposium on User interface software and technology
Proceedings of the VLDB Endowment
Collaboratively crowdsourcing workflows with turkomatic
Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work
CrowdWeaver: visually managing complex crowd work
Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work
Communications of the ACM
Human computation tasks with global constraints
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
CommunitySourcing: engaging local crowds to perform expert work via physical kiosks
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Socialproof: using crowdsourcing for correcting errors to improve speech based dictation experiences
CHI '12 Extended Abstracts on Human Factors in Computing Systems
So who won?: dynamic max discovery with the crowd
SIGMOD '12 Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data
A crowdsourcing quality control model for tasks distributed in parallel
CHI '12 Extended Abstracts on Human Factors in Computing Systems
Using real-time feedback to improve visual question answering
CHI '12 Extended Abstracts on Human Factors in Computing Systems
CrowdMine: towards crowdsourced human-assisted verification
Proceedings of the 49th Annual Design Automation Conference
Micro perceptual human computation for visual tasks
ACM Transactions on Graphics (TOG)
Medusa: a programming framework for crowd-sensing applications
Proceedings of the 10th international conference on Mobile systems, applications, and services
Real-time captioning by groups of non-experts
Proceedings of the 25th annual ACM symposium on User interface software and technology
Proceedings of the 3rd Annual ACM Web Science Conference
AutoMan: a platform for integrating human-based and digital computation
Proceedings of the ACM international conference on Object oriented programming systems languages and applications
Online quality control for real-time crowd captioning
Proceedings of the 14th international ACM SIGACCESS conference on Computers and accessibility
PKAW'12 Proceedings of the 12th Pacific Rim conference on Knowledge Management and Acquisition for Intelligent Systems
Knowledge acquisition and the rise of social machines
International Journal of Human-Computer Studies
CrowdLang: a programming language for the systematic exploration of human computation systems
SocInfo'12 Proceedings of the 4th international conference on Social Informatics
Proceedings of the 2013 conference on Computer supported cooperative work
Wally: crowd powered image matching on tablets
Proceedings of the First International Workshop on Crowdsourcing and Data Mining
Automated feedback generation for introductory programming assignments
Proceedings of the 34th ACM SIGPLAN conference on Programming language design and implementation
Leveraging transitive relations for crowdsourced joins
Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data
Cascade: crowdsourcing taxonomy creation
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Crowdsourcing performance evaluations of user interfaces
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
CrowdStudy: general toolkit for crowdsourced evaluation of web interfaces
Proceedings of the 5th ACM SIGCHI symposium on Engineering interactive computing systems
Statistical quality estimation for general crowdsourcing tasks
Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining
Chorus: a crowd-powered conversational assistant
Proceedings of the 26th annual ACM symposium on User interface software and technology
Answering visual questions with conversational crowd assistants
Proceedings of the 15th International ACM SIGACCESS Conference on Computers and Accessibility
The motivations and experiences of the on-demand mobile workforce
Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing
Information extraction and manipulation threats in crowd-powered systems
Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing
DeepCAPTCHA: an image CAPTCHA based on depth perception
Proceedings of the 5th ACM Multimedia Systems Conference
Crowdsourced Knowledge Acquisition: Towards Hybrid-Genre Workflows
International Journal on Semantic Web & Information Systems
| Hi-index | 0.02 |
Mechanical Turk (MTurk) provides an on-demand source of human computation. This provides a tremendous opportunity to explore algorithms which incorporate human computation as a function call. However, various systems challenges make this difficult in practice, and most uses of MTurk post large numbers of independent tasks. TurKit is a toolkit for prototyping and exploring algorithmic human computation, while maintaining a straight-forward imperative programming style. We present the crash-and-rerun programming model that makes TurKit possible, along with a variety of applications for human computation algorithms. We also present case studies of TurKit used for real experiments across different fields.