The interdisciplinary study of coordination
ACM Computing Surveys (CSUR)
The process recombinator: a tool for generating new business process ideas
ICIS '99 Proceedings of the 20th international conference on Information Systems
How can cooperative work tools support dynamic group process? bridging the specificity frontier
CSCW '00 Proceedings of the 2000 ACM conference on Computer supported cooperative work
BLEU: a method for automatic evaluation of machine translation
ACL '02 Proceedings of the 40th Annual Meeting on Association for Computational Linguistics
MapReduce: simplified data processing on large clusters
Communications of the ACM - 50th anniversary issue: 1958 - 2008
Exploring iterative and parallel human computation processes
Proceedings of the ACM SIGKDD Workshop on Human Computation
TurKit: human computation algorithms on mechanical turk
UIST '10 Proceedings of the 23nd annual ACM symposium on User interface software and technology
Soylent: a word processor with a crowd inside
UIST '10 Proceedings of the 23nd annual ACM symposium on User interface software and technology
Human computation: a survey and taxonomy of a growing field
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Platemate: crowdsourcing nutritional analysis from food photographs
Proceedings of the 24th annual ACM symposium on User interface software and technology
CrowdForge: crowdsourcing complex work
Proceedings of the 24th annual ACM symposium on User interface software and technology
The jabberwocky programming environment for structured social computing
Proceedings of the 24th annual ACM symposium on User interface software and technology
Communications of the ACM
Human computation tasks with global constraints
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Proceedings of the 3rd Annual ACM Web Science Conference
| Hi-index | 0.00 |
Human computation systems are often the result of extensive lengthy trial-and-error refinements. What we lack is an approach to systematically engineer solutions based on past successful patterns. In this paper we present the CrowdLang programming framework for engineering complex computation systems incorporating large crowds of networked humans and machines with a library of known interaction patterns. We evaluate CrowdLang by programming a German-to-English translation program incorporating machine translation and a monolingual crowd. The evaluation shows that CrowdLang is able to simply explore a large design space of possible problem-solving programs with the simple variation of the used abstractions. In an experiment involving 1918 different human actors, we show that the resulting translation program significantly outperforms a pure machine translation in terms of adequacy and fluency whilst translating more than 30 pages per hour and approximates the human-translated gold standard to 75%.