Analysis of inter-transcriber consistency in the Cat_ToBI prosodic labeling system

Authors:
David Escudero;Lourdes Aguilar;Maria del Mar Vanrell;Pilar Prieto
Affiliations:
Dpt. of Computer Sciences, Universidad de Valladolid, Spain;Dpt. of Spanish Philology, Universitat Autònoma de Barcelona, Spain;Dpt. of Catalan Philology, Universitat Autònoma de Barcelona, Spain and Dpt. of Translation and Language Sciences, Universitat Pompeu Fabra, Spain;Dpt. of Translation and Language Sciences, ICREA-Universitat Pompeu Fabra, Spain
Venue:
Speech Communication
Year:
2012

Citing 5
Cited 2

Automatic ToBI prediction and alignment to speed manual labeling of prosody

Speech Communication - Special issue on speech annotation and corpus tools
Classification of prosodic events using Quantized Contour Modeling

HLT '10 Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics
SWITCHBOARD: telephone speech corpus for research and development

ICASSP'92 Proceedings of the 1992 IEEE international conference on Acoustics, speech and signal processing - Volume 1
Automatic Prosodic Event Detection Using Acoustic, Lexical, and Syntactic Evidence

IEEE Transactions on Audio, Speech, and Language Processing
Exploiting Acoustic and Syntactic Features for Automatic Prosody Labeling in a Maximum Entropy Framework

IEEE Transactions on Audio, Speech, and Language Processing

A fuzzy classifier to deal with similarity between labels on automatic prosodic labeling

Computer Speech and Language
Glissando: a corpus for multidisciplinary prosodic studies in Spanish and Catalan

Language Resources and Evaluation

Quantified Score

Hi-index	0.00

Visualization

Abstract

A set of tools to analyze inconsistencies observed in a Cat_ToBI labeling experiment are presented. We formalize and use the metrics that are commonly used in inconsistency tests. The metrics are systematically applied to analyze the robustness of every symbol and every pair of transcribers. The results reveal agreement rates for this study that are comparable to previous ToBI inter-reliability tests. The inter-transcriber confusion rates are transformed into distance matrices to use multidimensional scaling for visualizing the confusion between the different ToBI symbols and the disagreement between the raters. Potential different labeling criteria are identified and subsets of symbols that are candidates to be fused are proposed.