Labeling images with a computer game
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Peekaboom: a game for locating objects in images
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Verbosity: a game for collecting common-sense facts
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Computer
Designing games with a purpose
Communications of the ACM - Designing games with a purpose
Matchin: eliciting user preferences with an online game
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
PhotoSlap: a multi-player online game for semantic annotation
AAAI'07 Proceedings of the 22nd national conference on Artificial intelligence - Volume 2
The role of game theory in human computation systems
Proceedings of the ACM SIGKDD Workshop on Human Computation
A crowdsourceable QoE evaluation framework for multimedia content
MM '09 Proceedings of the 17th ACM international conference on Multimedia
DevilTyper: a game for CAPTCHA usability evaluation
Computers in Entertainment (CIE) - Theoretical and Practical Computer Applications in Entertainment
Emerging theories and models of human computation systems: a brief survey
Proceedings of the 2nd international workshop on Ubiquitous crowdsouring
Capability-aligned matching: improving quality of games with a purpose
The 10th International Conference on Autonomous Agents and Multiagent Systems - Volume 2
A game-theoretic analysis of the ESP game
ACM Transactions on Economics and Computation - Inaugural Issue
Information verification during natural disasters
Proceedings of the 22nd international conference on World Wide Web companion
From sensing to controlling: the state of the art in ubiquitous crowdsourcing
International Journal of Communication Networks and Distributed Systems
| Hi-index | 0.00 |
The introduction of the ESP Game and other Games With A Purpose (GWAP) has demonstrated the potential of human computation in solving AI-hard problems. In such systems, users are normally required to input answers for questions proposed by the system, e.g., descriptions about a picture or a song. Since users may bring up irrelevant inputs intentionally or carelessly, and often the system does not have "correct" answers, we have to rely on the users to verify answers from others. We call this kind of mutual verification of users' answers "social verification." In this paper, we propose formal models for two fundamental social verification mechanisms, simultaneous verification and sequential verification, in human computation systems. By adopting a game-theoretic approach, we perform an equilibrium analysis which explains the effect of each verification mechanism on a system's outcome. Our analysis results show that sequential verification leads to a more diverse and descriptive set of outcomes than simultaneous verification, though the latter is stronger in ensuring the correctness of verified answers. Our experiments on Amazon Mechanical Turk, which asked users to input textual terms related to a word, confirmed our analysis results. We believe that our formal models for social verification mechanisms will provide a basis for the design of future human computation systems.