Dialogue segmentation with large numbers of volunteer internet annotators

Authors:
T. Daniel Midgley
Affiliations:
University of Western Australia, Perth, Australia
Venue:
ACL '09 Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2 - Volume 2
Year:
2009

Citing 9
Cited 0

Transformation-based error-driven learning and natural language processing: a case study in part-of-speech tagging

Computational Linguistics
Assessing agreement on classification tasks: the kappa statistic

Computational Linguistics
A critique and improvement of an evaluation metric for text segmentation

Computational Linguistics
A dialogue manager using initiative-response units and distributed control

EACL '91 Proceedings of the fifth conference on European chapter of the Association for Computational Linguistics
Dialogue act tagging with Transformation-Based Learning

COLING '98 Proceedings of the 17th international conference on Computational linguistics - Volume 2
Labeling images with a computer game

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Discourse segmentation of multi-party conversation

ACL '03 Proceedings of the 41st Annual Meeting on Association for Computational Linguistics - Volume 1
The Wisdom of Crowds

The Wisdom of Crowds
Unsupervised topic modelling for multi-party spoken discourse

ACL-44 Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics

Quantified Score

Hi-index	0.00

Visualization

Abstract

This paper shows the results of an experiment in dialogue segmentation. In this experiment, segmentation was done on a level of analysis similar to adjacency pairs. The method of annotation was somewhat novel: volunteers were invited to participate over the Web, and their responses were aggregated using a simple voting method. Though volunteers received a minimum of training, the aggregated responses of the group showed very high agreement with expert opinion. The group, as a unit, performed at the top of the list of annotators, and in many cases performed as well as or better than the best annotator.