Phrase detectives: Utilizing collective intelligence for internet-scale language resource creation

Authors:
Massimo Poesio;Jon Chamberlain;Udo Kruschwitz;Livio Robaldo;Luca Ducceschi
Affiliations:
University of Essex, UK;University of Essex, UK;University of Essex, UK;University of Turin, Italy;University of Utrecht/Verona, The Netherlands
Venue:
ACM Transactions on Interactive Intelligent Systems (TiiS) - Special section on internet-scale human problem solving and regular papers
Year:
2013

Citing 47
Cited 3

Symbiotic intelligence (poster): self-organizing knowledge on distributed networks driven by human interaction

ALIFE Proceedings of the sixth international conference on Artificial life
An empirically based system for processing definite descriptions

Computational Linguistics
Building a large annotated corpus of English: the penn treebank

Computational Linguistics - Special issue on using large corpora: II
A machine learning approach to coreference resolution of noun phrases

Computational Linguistics - Special issue on computational anaphora resolution
A corpus-based investigation of definite description use

Computational Linguistics
Long distance pronominalisation and global focus

COLING '98 Proceedings of the 17th international conference on Computational linguistics - Volume 1
Labeling images with a computer game

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
A model-theoretic coreference scoring scheme

MUC6 '95 Proceedings of the 6th conference on Message understanding
Improving the design of intelligent acquisition interfaces for collecting world knowledge from web contributors

Proceedings of the 3rd international conference on Knowledge capture
Collecting paraphrase corpora from volunteer contributors

Proceedings of the 3rd international conference on Knowledge capture
Peekaboom: a game for locating objects in images

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Games with a Purpose

Computer
A Theory of Fun for Game Design

A Theory of Fun for Game Design
Learning accurate, compact, and interpretable tree annotation

ACL-44 Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics
Mark-up Barking Up the Wrong Tree

Computational Linguistics
Twenty-five years of end-user searching, Part 1: Research findings

Journal of the American Society for Information Science and Technology
Unrestricted Coreference: Identifying Entities and Events in OntoNotes

ICSC '07 Proceedings of the International Conference on Semantic Computing
Semantic Wikipedia

Web Semantics: Science, Services and Agents on the World Wide Web
Games with a Purpose for the Semantic Web

IEEE Intelligent Systems
Designing games with a purpose

Communications of the ACM - Designing games with a purpose
Linking Documents to Encyclopedic Knowledge

IEEE Intelligent Systems
Inter-coder agreement for computational linguistics

Computational Linguistics
Crowdsourcing for relevance evaluation

ACM SIGIR Forum
Games with a purpose for social networking platforms

Proceedings of the 20th ACM conference on Hypertext and hypermedia
Towards methods for the collective gathering and quality control of relevance assessments

Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval
Discourse annotation and semantic annotation in the GNOME corpus

DiscAnnotation '04 Proceedings of the 2004 ACL Workshop on Discourse Annotation
Cheap and fast---but is it good?: evaluating non-expert annotations for natural language tasks

EMNLP '08 Proceedings of the Conference on Empirical Methods in Natural Language Processing
Unsupervised models for coreference resolution

EMNLP '08 Proceedings of the Conference on Empirical Methods in Natural Language Processing
OntoNotes: the 90% solution

NAACL-Short '06 Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers
Knowledge derived from wikipedia for computing semantic relatedness

Journal of Artificial Intelligence Research
Addressing the resource bottleneck to create large-scale annotated texts

STEP '08 Proceedings of the 2008 Conference on Semantics in Text Processing
Web-based annotation of anaphoric relations and lexical chains

LAW '07 Proceedings of the Linguistic Annotation Workshop
Play the language: play coreference

ACLShort '09 Proceedings of the ACL-IJCNLP 2009 Conference Short Papers
Distant supervision for relation extraction without labeled data

ACL '09 Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2 - Volume 2
Fast, cheap, and creative: evaluating translation quality using Amazon's Mechanical Turk

EMNLP '09 Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 1 - Volume 1
Acquiring high quality non-expert knowledge from on-demand workforce

People's Web '09 Proceedings of the 2009 Workshop on The People's Web Meets NLP: Collaboratively Constructed Semantic Resources
Constructing an anaphorically annotated corpus with non-experts: assessing the quality of collaborative annotations

People's Web '09 Proceedings of the 2009 Workshop on The People's Web Meets NLP: Collaboratively Constructed Semantic Resources
Financial incentives and the "performance of crowds"

ACM SIGKDD Explorations Newsletter
Motivations of Wikipedia content contributors

Computers in Human Behavior
SemEval-2010 task 1: Coreference resolution in multiple languages

SemEval '10 Proceedings of the 5th International Workshop on Semantic Evaluation
BART: A multilingual anaphora resolution system

SemEval '10 Proceedings of the 5th International Workshop on Semantic Evaluation
Report on the second NLG challenge on generating instructions in virtual environments (GIVE-2)

INLG '10 Proceedings of the 6th International Natural Language Generation Conference
In search of quality in crowdsourcing for search engine evaluation

ECIR'11 Proceedings of the 33rd European conference on Advances in information retrieval
Italian anaphoric annotation with the phrase detectives game-with-a-purpose

AI*IA'11 Proceedings of the 12th international conference on Artificial intelligence around man and beyond
Evaluation and user preference study on spatial diversity

ECIR'2010 Proceedings of the 32nd European conference on Advances in Information Retrieval
CoNLL-2011 shared task: modeling unrestricted coreference in OntoNotes

CONLL Shared Task '11 Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task
Sentence-Level attachment prediction

IRFC'10 Proceedings of the First international Information Retrieval Facility conference on Adbances in Multidisciplinary Retrieval

Crowdsourcing research opportunities: lessons from natural language processing

Proceedings of the 12th International Conference on Knowledge Management and Knowledge Technologies
Games with a Purpose or Mechanised Labour?: A Comparative Study

Proceedings of the 13th International Conference on Knowledge Management and Knowledge Technologies
Crowdsourced Knowledge Acquisition: Towards Hybrid-Genre Workflows

International Journal on Semantic Web & Information Systems

Quantified Score

Hi-index	0.00

Visualization

Abstract

We are witnessing a paradigm shift in Human Language Technology (HLT) that may well have an impact on the field comparable to the statistical revolution: acquiring large-scale resources by exploiting collective intelligence. An illustration of this new approach is Phrase Detectives, an interactive online game with a purpose for creating anaphorically annotated resources that makes use of a highly distributed population of contributors with different levels of expertise. The purpose of this article is to first of all give an overview of all aspects of Phrase Detectives, from the design of the game and the HLT methods we used to the results we have obtained so far. It furthermore summarizes the lessons that we have learned in developing this game which should help other researchers to design and implement similar games.