Using AUC and Accuracy in Evaluating Learning Algorithms

Authors:
Jin Huang;Charles X. Ling
Affiliations:
-;-
Venue:
IEEE Transactions on Knowledge and Data Engineering
Year:
2005

Citing 13
Cited 84

Signal detection theory: valuable tools for evaluating inductive learning

Proceedings of the sixth international workshop on Machine learning
A training algorithm for optimal margin classifiers

COLT '92 Proceedings of the fifth annual workshop on Computational learning theory
C4.5: programs for machine learning

C4.5: programs for machine learning
An introduction to support Vector Machines: and other kernel-based learning methods

An introduction to support Vector Machines: and other kernel-based learning methods
A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems

Machine Learning
A Tutorial on Support Vector Machines for Pattern Recognition

Data Mining and Knowledge Discovery
Discretization: An Enabling Technique

Data Mining and Knowledge Discovery
Learning Decision Trees Using the Area Under the ROC Curve

ICML '02 Proceedings of the Nineteenth International Conference on Machine Learning
The Case against Accuracy Estimation for Comparing Induction Algorithms

ICML '98 Proceedings of the Fifteenth International Conference on Machine Learning
Toward Bayesian Classifiers with Accurate Probabilities

PAKDD '02 Proceedings of the 6th Pacific-Asia Conference on Advances in Knowledge Discovery and Data Mining
Tree Induction for Probability-Based Ranking

Machine Learning
Learning to order things

Journal of Artificial Intelligence Research
The use of the area under the ROC curve in the evaluation of machine learning algorithms

Pattern Recognition

Preface: Special issue on "ROC Analysis in Pattern Recognition"

Pattern Recognition Letters - Special issue: ROC analysis in pattern recognition
Exploiting AUC for optimal linear combinations of dichotomizers

Pattern Recognition Letters - Special issue: ROC analysis in pattern recognition
Extracting Actionable Knowledge from Decision Trees

IEEE Transactions on Knowledge and Data Engineering
Metadata and its impact on libraries: Book Reviews

Journal of the American Society for Information Science and Technology
K-Means+ID3: A Novel Method for Supervised Anomaly Detection by Cascading K-Means Clustering and ID3 Decision Tree Learning Methods

IEEE Transactions on Knowledge and Data Engineering
Rule Extraction from Support Vector Machines: A Sequential Covering Approach

IEEE Transactions on Knowledge and Data Engineering
Domain-Driven, Actionable Knowledge Discovery

IEEE Intelligent Systems
Letters: Novelty detection with constructive probabilistic neural networks

Neurocomputing
Maximizing the area under the ROC curve by pairwise feature combination

Pattern Recognition
An adaptive anomaly detector for worm detection

SYSML'07 Proceedings of the 2nd USENIX workshop on Tackling computer systems problems with machine learning techniques
Detection of stock price movements using chance discovery and genetic programming

International Journal of Knowledge-based and Intelligent Engineering Systems - Chance discovery
Instance weighting versus threshold adjusting for cost-sensitive classification

Knowledge and Information Systems
When Overlapping Unexpectedly Alters the Class Imbalance Effects

IbPRIA '07 Proceedings of the 3rd Iberian conference on Pattern Recognition and Image Analysis, Part II
An Empirical Comparison of Ideal and Empirical ROC-Based Reject Rules

MLDM '07 Proceedings of the 5th international conference on Machine Learning and Data Mining in Pattern Recognition
Selection of Experts for the Design of Multiple Biometric Systems

MLDM '07 Proceedings of the 5th international conference on Machine Learning and Data Mining in Pattern Recognition
An Improved Model Selection Heuristic for AUC

ECML '07 Proceedings of the 18th European conference on Machine Learning
An experimental comparison of performance measures for classification

Pattern Recognition Letters
Incorporating domain knowledge into data mining classifiers: An application in indirect lending

Decision Support Systems
Support Vector Machine for Outlier Detection in Breast Cancer Survivability Prediction

Advanced Web and NetworkTechnologies, and Applications
A New Performance Evaluation Method for Two-Class Imbalanced Problems

SSPR & SPR '08 Proceedings of the 2008 Joint IAPR International Workshop on Structural, Syntactic, and Statistical Pattern Recognition
Designing multiple biometric systems: Measures of ensemble effectiveness

Engineering Applications of Artificial Intelligence
On the use of surrounding neighbors for synthetic over-sampling of the minority class

SMO'08 Proceedings of the 8th conference on Simulation, modelling and optimization
An efficient algorithm for learning to rank from preference graphs

Machine Learning
On modeling software defect repair time

Empirical Software Engineering
Efficient AUC Maximization with Regularized Least-Squares

Proceedings of the 2008 conference on Tenth Scandinavian Conference on Artificial Intelligence: SCAI 2008
Analysis of Time Series Novelty Detection Strategies for Synthetic and Real Data

Neural Processing Letters
Improving the Performance of Fuzzy Rule Based Classification Systems for Highly Imbalanced Data-Sets Using an Evolutionary Adaptive Inference System

IWANN '09 Proceedings of the 10th International Work-Conference on Artificial Neural Networks: Part I: Bio-Inspired Systems: Computational and Ambient Intelligence
Index of Balanced Accuracy: A Performance Measure for Skewed Class Distributions

IbPRIA '09 Proceedings of the 4th Iberian Conference on Pattern Recognition and Image Analysis
A First Study on the Use of Interval-Valued Fuzzy Sets with Genetic Tuning for Classification with Imbalanced Data-Sets

HAIS '09 Proceedings of the 4th International Conference on Hybrid Artificial Intelligence Systems
Toward breast cancer survivability prediction models through improving training space

Expert Systems with Applications: An International Journal
Predicting forensic admission among the mentally ill in a multinational setting: A Bayesian modelling approach

Data & Knowledge Engineering
A novel measure for evaluating classifiers

Expert Systems with Applications: An International Journal
Learning on class imbalanced data to classify peer-to-peer applications in IP traffic using resampling techniques

IJCNN'09 Proceedings of the 2009 international joint conference on Neural Networks
On the 2-tuples based genetic tuning performance for fuzzy rule based classification systems in imbalanced data-sets

Information Sciences: an International Journal
On the ability of complexity metrics to predict fault-prone classes in object-oriented systems

Journal of Systems and Software
Rasch-based high-dimensionality data reduction and class prediction with applications to microarray gene expression data

Expert Systems with Applications: An International Journal
Combining SVM classifiers using genetic fuzzy systems based on AUC for gene expression data analysis

ISBRA'07 Proceedings of the 3rd international conference on Bioinformatics research and applications
Index driven combination of multiple biometric experts for AUC maximisation

MCS'07 Proceedings of the 7th international conference on Multiple classifier systems
Iterative Boolean combination of classifiers in the ROC space: An application to anomaly detection with HMMs

Pattern Recognition
Distributed phishing detection by applying variable election using bayesian additive regression trees

ICC'09 Proceedings of the 2009 IEEE international conference on Communications
Fusion of fuzzy statistical distributions for classification of thyroid ultrasound patterns

Artificial Intelligence in Medicine
Defect prediction from static code features: current results, limitations, new approaches

Automated Software Engineering
Multi-class imbalanced data-sets with linguistic fuzzy rule based classification systems based on pairwise learning

IPMU'10 Proceedings of the Computational intelligence for knowledge-based systems design, and 13th international conference on Information processing and management of uncertainty
Dynamic linear combination of two-class classifiers

SSPR&SPR'10 Proceedings of the 2010 joint IAPR international conference on Structural, syntactic, and statistical pattern recognition
Combination of dichotomizers for maximizing the partial area under the ROC curve

SSPR&SPR'10 Proceedings of the 2010 joint IAPR international conference on Structural, syntactic, and statistical pattern recognition
Genetics-based machine learning for rule induction: state of the art, taxonomy, and comparative study

IEEE Transactions on Evolutionary Computation
An experimental comparison of cross-validation techniques for estimating the area under the ROC curve

Computational Statistics & Data Analysis
Regularized logistic regression without a penalty term: An application to cancer classification with microarray data

Expert Systems with Applications: An International Journal
An overview of ensemble methods for binary classifiers in multi-class problems: Experimental study on one-vs-one and one-vs-all schemes

Pattern Recognition
A dynamic over-sampling procedure based on sensitivity for multi-class problems

Pattern Recognition
Partial AUC maximization in a linear combination of dichotomizers

Pattern Recognition
A comparative analysis of methods for probability estimation tree

WSEAS Transactions on Computers
Addressing the classification with imbalanced data: open problems and new challenges on class distribution

HAIS'11 Proceedings of the 6th international conference on Hybrid artificial intelligent systems - Volume Part I
On the effectiveness of preprocessing methods when dealing with different levels of class imbalance

Knowledge-Based Systems
Adaptive ROC-based ensembles of HMMs applied to anomaly detection

Pattern Recognition
Selection strategies for pAUC-based combination of dichotomizers

MCS'11 Proceedings of the 10th international conference on Multiple classifier systems
An empirical comparison of flat and hierarchical performance measures for multi-label classification with hierarchy extraction

KES'11 Proceedings of the 15th international conference on Knowledge-based and intelligent information and engineering systems - Volume Part I
Study on preprocessing and classifying mass spectral raw data concerning human normal and disease cases

ISBMDA'06 Proceedings of the 7th international conference on Biological and Medical Data Analysis
AUC-Based linear combination of dichotomizers

SSPR'06/SPR'06 Proceedings of the 2006 joint IAPR international conference on Structural, Syntactic, and Statistical Pattern Recognition
Analysis of feature rankings for classification

IDA'05 Proceedings of the 6th international conference on Advances in Intelligent Data Analysis
Analysis of preprocessing vs. cost-sensitive learning for imbalanced classification. Open problems on intrinsic data characteristics

Expert Systems with Applications: An International Journal
A two-stage evolutionary algorithm based on sensitivity and accuracy for multi-class problems

Information Sciences: an International Journal
Learning and evaluation in the presence of class hierarchies: application to text categorization

AI'06 Proceedings of the 19th international conference on Advances in Artificial Intelligence: Canadian Society for Computational Studies of Intelligence
A Robust Multi-Class Feature Selection Strategy Based on Rotation Forest Ensemble Algorithm for Diagnosis of Erythemato-Squamous Diseases

Journal of Medical Systems
Feature selection for MAUC-oriented classification systems

Neurocomputing
Including spatial interdependence in customer acquisition models: A cross-category comparison

Expert Systems with Applications: An International Journal
Masquerade attacks based on user's profile

Journal of Systems and Software
Predicting noise filtering efficacy with data complexity measures for nearest neighbor classification

Pattern Recognition
Optimized feature extraction and actionable knowledge discovery for Customer Relationship Management (CRM)

Proceedings of the Second International Conference on Computational Science, Engineering and Information Technology
Fuzzy similarity-based nearest-neighbour classification as alternatives to their fuzzy-rough parallels

International Journal of Approximate Reasoning
Building a generic graph-based descriptor set for use in drug discovery

AusDM '09 Proceedings of the Eighth Australasian Data Mining Conference - Volume 101
Acquaintance or partner?: predicting partnership in online and location-based social networks

Proceedings of the 2013 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining
Customer attrition in retailing: An application of Multivariate Adaptive Regression Splines

Expert Systems with Applications: An International Journal
Spatial Recurrences for Pedestrian Classification

Journal of Mathematical Imaging and Vision
EUSBoost: Enhancing ensembles for highly imbalanced data-sets by evolutionary undersampling

Pattern Recognition
Texture classification using kernel-based techniques

IWANN'13 Proceedings of the 12th international conference on Artificial Neural Networks: advances in computational intelligence - Volume Part I
Classification and outlier detection based on topic based pattern synthesis

MLDM'13 Proceedings of the 9th international conference on Machine Learning and Data Mining in Pattern Recognition
An improved method of early diagnosis of smoking-induced respiratory changes using machine learning algorithms

Computer Methods and Programs in Biomedicine
Addressing imbalanced classification with instance generation techniques: IPADE-ID

Neurocomputing
On the importance of the validation technique for classification with imbalanced datasets: Addressing covariate shift when data is skewed

Information Sciences: an International Journal
A novel method for combining Bayesian networks, theoretical analysis, and its applications

Pattern Recognition
Multi-label learning under feature extraction budgets

Pattern Recognition Letters
Comprehensible classification models: a position paper

ACM SIGKDD Explorations Newsletter
Sharpened graph ensemble for semi-supervised learning

Intelligent Data Analysis

Quantified Score

Hi-index	0.01

Visualization

Abstract

The area under the ROC (Receiver Operating Characteristics) curve, or simply AUC, has been traditionally used in medical diagnosis since the 1970s. It has recently been proposed as an alternative single-number measure for evaluating the predictive ability of learning algorithms. However, no formal arguments were given as to why AUC should be preferred over accuracy. In this paper, we establish formal criteria for comparing two different measures for learning algorithms and we show theoretically and empirically that AUC is a better measure (defined precisely) than accuracy. We then reevaluate well-established claims in machine learning based on accuracy using AUC and obtain interesting and surprising new results. For example, it has been well-established and accepted that Naive Bayes and decision trees are very similar in predictive accuracy. We show, however, that Naive Bayes is significantly better than decision trees in AUC. The conclusions drawn in this paper may make a significant impact on machine learning and data mining applications.