Constructing an anaphorically annotated corpus with non-experts: assessing the quality of collaborative annotations

Authors:
Jon Chamberlain;Udo Kruschwitz;Massimo Poesio
Affiliations:
University of Essex;University of Essex;University of Essex
Venue:
People's Web '09 Proceedings of the 2009 Workshop on The People's Web Meets NLP: Collaboratively Constructed Semantic Resources
Year:
2009

Citing 11
Cited 6

Symbiotic intelligence (poster): self-organizing knowledge on distributed networks driven by human interaction

ALIFE Proceedings of the sixth international conference on Artificial life
The Wisdom of Crowds

The Wisdom of Crowds
Peekaboom: a game for locating objects in images

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Games with a Purpose

Computer
Speech and Language Processing (2nd Edition)

Speech and Language Processing (2nd Edition)
Semantic Wikipedia

Web Semantics: Science, Services and Agents on the World Wide Web
Learning consensus opinion: mining data from a labeling game

Proceedings of the 18th international conference on World wide web
Playful tagging: folksonomy generation using online games

Proceedings of the 18th international conference on World wide web
Discourse annotation and semantic annotation in the GNOME corpus

DiscAnnotation '04 Proceedings of the 2004 ACL Workshop on Discourse Annotation
OntoNotes: the 90% solution

NAACL-Short '06 Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers
Web-based annotation of anaphoric relations and lexical chains

LAW '07 Proceedings of the Linguistic Annotation Workshop

PackPlay: mining semantic data in collaborative games

LAW IV '10 Proceedings of the Fourth Linguistic Annotation Workshop
Italian anaphoric annotation with the phrase detectives game-with-a-purpose

AI*IA'11 Proceedings of the 12th international conference on Artificial intelligence around man and beyond
Collective intelligence and language resources: introduction to the special issue on collaboratively constructed language resources

Language Resources and Evaluation
Supervised collaboration for syntactic annotation of Quranic Arabic

Language Resources and Evaluation
Perspectives on crowdsourcing annotations for natural language processing

Language Resources and Evaluation
Phrase detectives: Utilizing collective intelligence for internet-scale language resource creation

ACM Transactions on Interactive Intelligent Systems (TiiS) - Special section on internet-scale human problem solving and regular papers

Quantified Score

Hi-index	0.00

Visualization

Abstract

This paper reports on the ongoing work of Phrase Detectives, an attempt to create a very large anaphorically annotated text corpus. Annotated corpora of the size needed for modern computational linguistics research cannot be created by small groups of hand-annotators however the ESP game and similar games with a purpose have demonstrated how it might be possible to do this through Web collaboration. We show that this approach could be used to create large, high-quality natural language resources.