Who are the crowdworkers?: shifting demographics in mechanical turk

Authors:
Joel Ross;Lilly Irani;M. Six Silberman;Andrew Zaldivar;Bill Tomlinson
Affiliations:
University of California, Irvine, Irvine, CA, USA;University of California, Irvine, Irvine, CA, USA;University of California, Irvine, Irvine, CA, USA;University of California, Irvine, Irvine, CA, USA;University of California, Irvine, Irvine, CA, USA
Venue:
CHI '10 Extended Abstracts on Human Factors in Computing Systems
Year:
2010

Citing 8
Cited 67

AI Gets a Brain

Queue - AI
Human computation

Proceedings of the 4th international conference on Knowledge capture
Crowdsourcing user studies with Mechanical Turk

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Designing games with a purpose

Communications of the ACM - Designing games with a purpose
Towards a model of understanding social search

Proceedings of the 2008 ACM conference on Computer supported cooperative work
Crowdsourcing for relevance evaluation

ACM SIGIR Forum
Cheap and fast---but is it good?: evaluating non-expert annotations for natural language tasks

EMNLP '08 Proceedings of the Conference on Empirical Methods in Natural Language Processing
An exploration into activity-informed physical advertising using PEST

PERVASIVE'07 Proceedings of the 5th international conference on Pervasive computing

Modulating video credibility via visualization of quality evaluations

Proceedings of the 4th workshop on Information credibility
Sellers' problems in human computation markets

Proceedings of the ACM SIGKDD Workshop on Human Computation
Exploring the use of crowdsourcing to support empirical studies in software engineering

Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement
Soylent: a word processor with a crowd inside

UIST '10 Proceedings of the 23nd annual ACM symposium on User interface software and technology
Analyzing the Amazon Mechanical Turk marketplace

XRDS: Crossroads, The ACM Magazine for Students - Comp-YOU-Ter
Ethics and tactics of professional crowdwork

XRDS: Crossroads, The ACM Magazine for Students - Comp-YOU-Ter
Efficiently scaling up video annotation with crowdsourced marketplaces

ECCV'10 Proceedings of the 11th European conference on Computer vision: Part IV
Evaluating and improving the usability of Mechanical Turk for low-income workers in India

Proceedings of the First ACM Symposium on Computing for Development
Turkalytics: analytics for human computation

Proceedings of the 20th international conference on World wide web
Affective computational priming and creativity

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Cooks or cobblers?: crowd creativity through combination

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
The impact of social information on visual judgments

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
An examination of two delivery modes for interactive search system experiments: remote and laboratory

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Utility of human-computer interactions: toward a science of preference measurement

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Web workers unite! addressing challenges of online laborers

CHI '11 Extended Abstracts on Human Factors in Computing Systems
Toward sustainable software engineering (NIER track)

Proceedings of the 33rd International Conference on Software Engineering
CrowdDB: answering queries with crowdsourcing

Proceedings of the 2011 ACM SIGMOD International Conference on Management of data
Personifying programming tool feedback improves novice programmers' learning

Proceedings of the seventh international workshop on Computing education research
Putting out a HIT: crowdsourcing malware installs

WOOT'11 Proceedings of the 5th USENIX conference on Offensive technologies
Improving computer security dialogs

INTERACT'11 Proceedings of the 13th IFIP TC 13 international conference on Human-computer interaction - Volume Part IV
Modeling of crowdsourcing platforms and granularity of work organization in future internet

Proceedings of the 23rd International Teletraffic Congress
Worker types and personality traits in crowdsourcing relevance labels

Proceedings of the 20th ACM international conference on Information and knowledge management
Collaborative information seeking by the numbers

Proceedings of the 3rd international workshop on Collaborative information retrieval
Perception of personality and naturalness through dialogues by native speakers of American English and Arabic

SIGDIAL '11 Proceedings of the SIGDIAL 2011 Conference
Vice interfaces

Proceedings of the Sixth International Conference on Tangible, Embedded and Embodied Interaction
Serf and turf: crowdturfing for fun and profit

Proceedings of the 21st international conference on World Wide Web
mClerk: enabling mobile crowdsourcing in developing regions

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Comparing averages in time series data

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
CommunitySourcing: engaging local crowds to perform expert work via physical kiosks

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Social desirability bias and self-reports of motivation: a study of amazon mechanical turk in the US and India

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Effective temporal graph layout: a comparative study of animation versus static display methods

Information Visualization
Learning from crowds in the presence of schools of thought

Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining
Opening "open" innovation

DPPI '11 Proceedings of the 2011 Conference on Designing Pleasurable Products and Interfaces
Clickjacking: attacks and defenses

Security'12 Proceedings of the 21st USENIX conference on Security symposium
Paying in kind for crowdsourced work in developing regions

Pervasive'12 Proceedings of the 10th international conference on Pervasive Computing
What do online behavioral advertising privacy disclosures communicate to users?

Proceedings of the 2012 ACM workshop on Privacy in the electronic society
Crowdsourcing micro-level multimedia annotations: the challenges of evaluation and interface

Proceedings of the ACM multimedia 2012 workshop on Crowdsourcing for multimedia
The face of quality in crowdsourcing relevance labels: demographics, personality and labeling accuracy

Proceedings of the 21st ACM international conference on Information and knowledge management
GeoCrowd: enabling query answering with spatial crowdsourcing

Proceedings of the 20th International Conference on Advances in Geographic Information Systems
Efficiently Scaling up Crowdsourced Video Annotation

International Journal of Computer Vision
The future of crowd work

Proceedings of the 2013 conference on Computer supported cooperative work
Auction-based crowdsourcing supporting skill management

Information Systems
How to filter out random clickers in a crowdsourcing-based study?

Proceedings of the 2012 BELIV Workshop: Beyond Time and Errors - Novel Evaluation Methods for Visualization
An internet-scale idea generation system

ACM Transactions on Interactive Intelligent Systems (TiiS) - Special section on internet-scale human problem solving and regular papers
An introduction to crowdsourcing for language and multimedia technology research

PROMISE'12 Proceedings of the 2012 international conference on Information Retrieval Meets Information Visualization
Challenges and Opportunities for Trust Management in Crowdsourcing

WI-IAT '12 Proceedings of the The 2012 IEEE/WIC/ACM International Joint Conferences on Web Intelligence and Intelligent Agent Technology - Volume 02
Crowdsourcing interactions: using crowdsourcing for evaluating interactive information retrieval systems

Information Retrieval
Crowdsourcing for information retrieval: introduction to the special issue

Information Retrieval
Increasing cheat robustness of crowdsourcing tasks

Information Retrieval
An analysis of human factors and label accuracy in crowdsourcing relevance judgments

Information Retrieval
Tagging human activities in video by crowdsourcing

Proceedings of the 3rd ACM conference on International conference on multimedia retrieval
Experiences surveying the crowd: reflections on methods, participation, and reliability

Proceedings of the 5th Annual ACM Web Science Conference
"i read my Twitter the next morning and was astonished": a conversational perspective on Twitter regrets

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Priming creativity through improvisation on an adaptive musical instrument

Proceedings of the 9th ACM Conference on Creativity & Cognition
Turkopticon: interrupting worker invisibility in amazon mechanical turk

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Labor dynamics in a mobile micro-task market

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Quantity estimation in visualizations of tagged text

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
In-game assessments increase novice programmers' engagement and level completion speed

Proceedings of the ninth annual international ACM conference on International computing education research
Your attention please: designing security-decision UIs to make genuine risks harder to ignore

Proceedings of the Ninth Symposium on Usable Privacy and Security
What matters to users?: factors that affect users' willingness to share information with online advertisers

Proceedings of the Ninth Symposium on Usable Privacy and Security
Feasibility of identifying eating moments from first-person images leveraging human computation

Proceedings of the 4th International SenseCam & Pervasive Imaging Conference
Being a turker

Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing
The motivations and experiences of the on-demand mobile workforce

Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing
A comparison of social, learning, and financial strategies on crowd engagement and output quality

Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing
A reputation management approach for resource constrained trustee agents

IJCAI'13 Proceedings of the Twenty-Third international joint conference on Artificial Intelligence
Toward crowdsourcing micro-level behavior annotations: the challenges of interface, training, and generalization

Proceedings of the 19th international conference on Intelligent User Interfaces
The wisdom of minority: discovering and targeting the right group of workers for crowdsourcing

Proceedings of the 23rd international conference on World wide web

Quantified Score

Hi-index	0.00

Visualization

Abstract

Amazon Mechanical Turk (MTurk) is a crowdsourcing system in which tasks are distributed to a population of thousands of anonymous workers for completion. This system is increasingly popular with researchers and developers. Here we extend previous studies of the demographics and usage behaviors of MTurk workers. We describe how the worker population has changed over time, shifting from a primarily moderate-income, U.S.-based workforce towards an increasingly international group with a significant population of young, well-educated Indian workers. This change in population points to how workers may treat Turking as a full-time job, which they rely on to make ends meet.