The interdisciplinary study of coordination
ACM Computing Surveys (CSUR)
The process recombinator: a tool for generating new business process ideas
ICIS '99 Proceedings of the 20th international conference on Information Systems
How can cooperative work tools support dynamic group process? bridging the specificity frontier
CSCW '00 Proceedings of the 2000 ACM conference on Computer supported cooperative work
MapReduce: simplified data processing on large clusters
Communications of the ACM - 50th anniversary issue: 1958 - 2008
TurKit: human computation algorithms on mechanical turk
UIST '10 Proceedings of the 23nd annual ACM symposium on User interface software and technology
Soylent: a word processor with a crowd inside
UIST '10 Proceedings of the 23nd annual ACM symposium on User interface software and technology
Platemate: crowdsourcing nutritional analysis from food photographs
Proceedings of the 24th annual ACM symposium on User interface software and technology
CrowdForge: crowdsourcing complex work
Proceedings of the 24th annual ACM symposium on User interface software and technology
The jabberwocky programming environment for structured social computing
Proceedings of the 24th annual ACM symposium on User interface software and technology
Communications of the ACM
Human computation tasks with global constraints
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
CrowdLang: a programming language for the systematic exploration of human computation systems
SocInfo'12 Proceedings of the 4th international conference on Social Informatics
Proceedings of the 22nd international conference on World Wide Web
An introduction to human computation and games with a purpose
ICWE'13 Proceedings of the 13th international conference on Web Engineering
| Hi-index | 0.00 |
In this paper we present the programming language and framework CrowdLang for engineering complex computation systems incorporating large numbers of networked humans and machines agents. We evaluate CrowdLang by developing a text translation program incorporating human and machine agents. The evaluation shows that we are able to simply explore a large design space of possible problem solving programs with the simple variation of the used abstractions. Furthermore, an experiment, involving 1918 different human actors, shows that the developed mixed human-machine translation program significantly outperforms a pure machine translation in terms of adequacy and fluency whilst translating more than 30 pages per hour and that the program approximates the professional translated gold-standard to 75% using the automatic evaluation metric METEOR. Last but not least, our evaluation illustrates that our new human computation pattern staged-contest with pruning outperforms all other refinements in the translation task.