A novel measure for evaluating classifiers

Authors:
Jin-Mao Wei;Xiao-Jie Yuan;Qing-Hua Hu;Shu-Qin Wang
Affiliations:
Department of Computer Science, Nankai University, Tianjin 300071, China;Department of Computer Science, Nankai University, Tianjin 300071, China;Harbin Institute of Technology, Harbin, China;College of Computer and Information Engineering, Tianjin Normal University, Tianjin 300387, China
Venue:
Expert Systems with Applications: An International Journal
Year:
2010

Citing 11
Cited 1

C4.5: programs for machine learning

C4.5: programs for machine learning
A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems

Machine Learning
Induction of Decision Trees

Machine Learning
No Unbiased Estimator of the Variance of K-Fold Cross-Validation

The Journal of Machine Learning Research
Using AUC and Accuracy in Evaluating Learning Algorithms

IEEE Transactions on Knowledge and Data Engineering
A comprehensive evaluation of multicategory classification methods for microarray gene expression cancer diagnosis

Bioinformatics
Multi-class ROC analysis from a multi-objective optimisation perspective

Pattern Recognition Letters - Special issue: ROC analysis in pattern recognition
Efficient Multiclass ROC Approximation by Decomposition via Confusion Matrix Perturbation Analysis

IEEE Transactions on Pattern Analysis and Machine Intelligence
A study of cross-validation and bootstrap for accuracy estimation and model selection

IJCAI'95 Proceedings of the 14th international joint conference on Artificial intelligence - Volume 2
The use of the area under the ROC curve in the evaluation of machine learning algorithms

Pattern Recognition
Confusion matrix disagreement for multiple classifiers

CIARP'07 Proceedings of the Congress on pattern recognition 12th Iberoamerican conference on Progress in pattern recognition, image analysis and applications

Pacc - a discriminative and accuracy correlated measure for assessment of classification results

MLDM'13 Proceedings of the 9th international conference on Machine Learning and Data Mining in Pattern Recognition

Quantified Score

Hi-index	12.05

Visualization

Abstract

Evaluating classifier performances is a crucial problem in pattern recognition and machine learning. In this paper, we propose a new measure, i.e. confusion entropy, for evaluating classifiers. For each class cl"i of an (N+1)-class problem, the misclassification information involves both the information of how the samples with true class label cl"i have been misclassified to the other N classes and the information of how the samples of the other N classes have been misclassified to class cl"i. The proposed measure exploits the class distribution information of such misclassifications of all classes. Both theoretical analysis and statistical experiments show the proposed measure is more precise than accuracy and RCI. Experimental results on some benchmark data sets further confirm the theoretical analysis and statistical results and show that the new measure is feasible for evaluating classifier performances.