Electronic peer review and peer grading in computer-science courses
Proceedings of the thirty-second SIGCSE technical symposium on Computer Science Education
Public Access Systems: Bringing Computer Power to the People
Public Access Systems: Bringing Computer Power to the People
Dynamo: a public interactive surface supporting the cooperative sharing and exchange of media
Proceedings of the 16th annual ACM symposium on User interface software and technology
Kimono: kiosk-mobile phone knowledge sharing system
MUM '05 Proceedings of the 4th international conference on Mobile and ubiquitous multimedia
BingBee, an information kiosk for social enablement in marginalized communities
SAICSIT '06 Proceedings of the 2006 annual research conference of the South African institute of computer scientists and information technologists on IT research in developing countries
Crowdsourcing user studies with Mechanical Turk
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Trustworthy and personalized computing on public kiosks
Proceedings of the 6th international conference on Mobile systems, applications, and services
Designing games with a purpose
Communications of the ACM - Designing games with a purpose
Proceedings of the ACM 2009 international conference on Supporting group work
Crowdsourcing graphical perception: using mechanical turk to assess visualization design
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
MOSES: exploring new ground in media and post-conflict reconciliation
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Who are the crowdworkers?: shifting demographics in mechanical turk
CHI '10 Extended Abstracts on Human Factors in Computing Systems
CrowdSearch: exploiting crowds for accurate real-time image search on mobile phones
Proceedings of the 8th international conference on Mobile systems, applications, and services
TurKit: human computation algorithms on mechanical turk
UIST '10 Proceedings of the 23nd annual ACM symposium on User interface software and technology
Soylent: a word processor with a crowd inside
UIST '10 Proceedings of the 23nd annual ACM symposium on User interface software and technology
VizWiz: nearly real-time answers to visual questions
UIST '10 Proceedings of the 23nd annual ACM symposium on User interface software and technology
Analyzing the Amazon Mechanical Turk marketplace
XRDS: Crossroads, The ACM Magazine for Students - Comp-YOU-Ter
Human computation: a survey and taxonomy of a growing field
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Creek watch: pairing usefulness and usability for successful citizen science
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Design lessons from the fastest q&a site in the west
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
CrowdForge: crowdsourcing complex work
CHI '11 Extended Abstracts on Human Factors in Computing Systems
Crowds in two seconds: enabling realtime crowd-powered interfaces
Proceedings of the 24th annual ACM symposium on User interface software and technology
The jabberwocky programming environment for structured social computing
Proceedings of the 24th annual ACM symposium on User interface software and technology
Human Computation
Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work Companion
Crowd vs. crowd: large-scale cooperative design through open team competition
Proceedings of the 2013 conference on Computer supported cooperative work
Proceedings of the 2013 conference on Computer supported cooperative work
Facilitating students' collaboration and learning in a question and answer system
Proceedings of the 2013 conference on Computer supported cooperative work companion
Collabmap: crowdsourcing maps for emergency planning
Proceedings of the 5th Annual ACM Web Science Conference
Cascade: crowdsourcing taxonomy creation
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Crowdsourcing user interface adaptations for minimizing the bloat in enterprise applications
Proceedings of the 5th ACM SIGCHI symposium on Engineering interactive computing systems
Cobi: a community-informed conference scheduling tool
Proceedings of the 26th annual ACM symposium on User interface software and technology
Sharing Knowledge and Expertise: The CSCW View of Knowledge Management
Computer Supported Cooperative Work
Scaling short-answer grading by combining peer assessment with algorithmic scoring
Proceedings of the first ACM conference on Learning @ scale conference
Hi-index | 0.01 |
Online labor markets, such as Amazon's Mechanical Turk, have been used to crowdsource simple, short tasks like image labeling and transcription. However, expert knowledge is often lacking in such markets, making it impossible to complete certain classes of tasks. In this work we introduce an alternative mechanism for crowdsourcing tasks that require specialized knowledge or skill: communitysourcing --- the use of physical kiosks to elicit work from specific populations. We investigate the potential of communitysourcing by designing, implementing and evaluating Umati: the communitysourcing vending machine. Umati allows users to earn credits by performing tasks using a touchscreen attached to the machine. Physical rewards (in this case, snacks) are dispensed through traditional vending mechanics. We evaluated whether communitysourcing can accomplish expert work by using Umati to grade Computer Science exams. We placed Umati in a university Computer Science building, targeting students with grading tasks for snacks. Over one week, 328 unique users (302 of whom were students) completed 7771 tasks (7240 by students). 80% of users had never participated in a crowdsourcing market before. We found that Umati was able to grade exams with 2% higher accuracy (at the same price) or at 33% lower cost (at equivalent accuracy) than traditional single-expert grading. Mechanical Turk workers had no success grading the same exams. These results indicate that communitysourcing can successfully elicit high-quality expert work from specific communities.