Generalized agreement statistics over fixed group of experts

Authors:
Mohak Shah
Affiliations:
Accenture Technology Labs, Chicago, IL
Venue:
ECML PKDD'11 Proceedings of the 2011 European conference on Machine learning and knowledge discovery in databases - Volume Part III
Year:
2011

Citing 4
Cited 0

Combining Pattern Classifiers: Methods and Algorithms

Combining Pattern Classifiers: Methods and Algorithms
Cheap and fast---but is it good?: evaluating non-expert annotations for natural language tasks

EMNLP '08 Proceedings of the Conference on Empirical Methods in Natural Language Processing
Learning From Crowds

The Journal of Machine Learning Research
Evaluating Learning Algorithms: A Classification Perspective

Evaluating Learning Algorithms: A Classification Perspective

Quantified Score

Hi-index	0.00

Visualization

Abstract

Generalizations of chance corrected statistics tomeasure interexpert agreement on class label assignments to the data instances have traditionally relied on the marginalization argument over a variable group of experts. Further, this argument has also resulted in agreement measures to evaluate the class predictions by an isolated classifier against the (multiple) labels assigned by the group of experts. We show that these measures are not necessarily suitable for application in the more typical fixed experts' group scenario. We also propose novel, moremeaningful, less variable generalizations for quantifying both the inter-expert agreement over the fixed group and assessing a classifier's output against it in a multiexpert multi-class scenario by taking into account expert-specific biases and correlations.