TurKit: tools for iterative tasks on mechanical Turk

Authors:
Greg Little;Lydia B. Chilton;Max Goldman;Robert C. Miller
Affiliations:
MIT CSAIL, Cambridge, MA;MIT CSAIL, Cambridge, MA;MIT CSAIL, Cambridge, MA;MIT CSAIL, Cambridge, MA
Venue:
Proceedings of the ACM SIGKDD Workshop on Human Computation
Year:
2009

Citing 2
Cited 36

Games with a Purpose

Computer
Harnessing the wisdom of crowds in wikipedia: quality through coordination

Proceedings of the 2008 ACM conference on Computer supported cooperative work

Exploring iterative and parallel human computation processes

Proceedings of the ACM SIGKDD Workshop on Human Computation
Toward automatic task design: a progress report

Proceedings of the ACM SIGKDD Workshop on Human Computation
Exploring the use of crowdsourcing to support empirical studies in software engineering

Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement
Creating speech and language data with Amazon's Mechanical Turk

CSLDAMT '10 Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk
Clustering dictionary definitions using Amazon Mechanical Turk

CSLDAMT '10 Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk
Collecting image annotations using Amazon's Mechanical Turk

CSLDAMT '10 Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk
Crowdsourcing human-based computation

Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries
Turkalytics: analytics for human computation

Proceedings of the 20th international conference on World wide web
A crowdsourcing model for receiving design critique

CHI '11 Extended Abstracts on Human Factors in Computing Systems
Shepherding the crowd: managing and providing feedback to crowd workers

CHI '11 Extended Abstracts on Human Factors in Computing Systems
Turkomatic: automatic recursive task and workflow design for mechanical turk

CHI '11 Extended Abstracts on Human Factors in Computing Systems
CrowdForge: crowdsourcing complex work

CHI '11 Extended Abstracts on Human Factors in Computing Systems
Tweetflows: flexible workflows with twitter

Proceedings of the 3rd International Workshop on Principles of Engineering Service-Oriented Systems
CrowdDB: answering queries with crowdsourcing

Proceedings of the 2011 ACM SIGMOD International Conference on Management of data
Execution control for crowdsourcing

Proceedings of the 24th annual ACM symposium adjunct on User interface software and technology
The jabberwocky programming environment for structured social computing

Proceedings of the 24th annual ACM symposium on User interface software and technology
Collaborative workflow for crowdsourcing translation

Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work
Sentence-Level attachment prediction

IRFC'10 Proceedings of the First international Information Retrieval Facility conference on Adbances in Multidisciplinary Retrieval
Strategies for crowdsourcing social data analysis

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Socially computed scripts to support social problem solving skills

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
CrowdScreen: algorithms for filtering data with humans

SIGMOD '12 Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data
Creating mobile ad hoc workflows with Twitter

Proceedings of the 27th Annual ACM Symposium on Applied Computing
Reducing the barriers to writing verified specifications

Proceedings of the ACM international conference on Object oriented programming systems languages and applications
CSAF: a community-sourcing annotation framework

LAW VI '12 Proceedings of the Sixth Linguistic Annotation Workshop
Combining human and computation intelligence: the case of data interlinking tools

International Journal of Metadata, Semantics and Ontologies
Deco: declarative crowdsourcing

Proceedings of the 21st ACM international conference on Information and knowledge management
CrowdMap: crowdsourcing ontology alignment with microtasks

ISWC'12 Proceedings of the 11th international conference on The Semantic Web - Volume Part I
Architecture-Driven modeling of adaptive collaboration structures in large-scale social web applications

WISE'12 Proceedings of the 13th international conference on Web Information Systems Engineering
Enhancing reliability using peer consistency evaluation in human computation

Proceedings of the 2013 conference on Computer supported cooperative work
Crowd vs. crowd: large-scale cooperative design through open team competition

Proceedings of the 2013 conference on Computer supported cooperative work
Increasing cheat robustness of crowdsourcing tasks

Information Retrieval
Incentives and rewarding in social computing

Communications of the ACM
Reactive crowdsourcing

Proceedings of the 22nd international conference on World Wide Web
Combining human and computation intelligence: the case of data interlinking tools

International Journal of Metadata, Semantics and Ontologies
An introduction to human computation and games with a purpose

ICWE'13 Proceedings of the 13th international conference on Web Engineering
Being a turker

Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing

Quantified Score

Hi-index	0.02

Visualization

Abstract

Mechanical Turk (MTurk) is an increasingly popular web service for paying people small rewards to do human computation tasks. Current uses of MTurk typically post independent parallel tasks. We are exploring an alternative iterative paradigm, in which workers build on or evaluate each other's work. We describe TurKit, a new toolkit for deploying iterative tasks to MTurk, with a familiar imperative programming paradigm that effectively uses MTurk workers as subroutines.