Cost curves: An improved method for visualizing classifier performance

Authors:
Chris Drummond;Robert C. Holte
Affiliations:
Institute for Information Technology, National Research Council Canada, Ontario, Canada K1A 0R6;Department of Computing Science, University of Alberta, Edmonton, Canada T6G 2E8
Venue:
Machine Learning
Year:
2006

Citing 29
Cited 46

Computational geometry: an introduction

Computational geometry: an introduction
C4.5: programs for machine learning

C4.5: programs for machine learning
Very Simple Classification Rules Perform Well on Most Commonly Used Datasets

Machine Learning
Learning in the “Real World”

Machine Learning - Special issue on applications of machine learning and the knowledge discovery process
Machine Learning for the Detection of Oil Spills in Satellite Radar Images

Machine Learning - Special issue on applications of machine learning and the knowledge discovery process
Robust classification systems for imprecise environments

AAAI '98/IAAI '98 Proceedings of the fifteenth national/tenth conference on Artificial intelligence/Innovative applications of artificial intelligence
MetaCost: a general method for making classifiers cost-sensitive

KDD '99 Proceedings of the fifth ACM SIGKDD international conference on Knowledge discovery and data mining
Explicitly representing expected cost: an alternative to ROC representation

Proceedings of the sixth ACM SIGKDD international conference on Knowledge discovery and data mining
Fast implementation of depth contours using topological sweep

SODA '01 Proceedings of the twelfth annual ACM-SIAM symposium on Discrete algorithms
Robust Classification for Imprecise Environments

Machine Learning
Information Retrieval

Information Retrieval
Adaptive Fraud Detection

Data Mining and Knowledge Discovery
The CN2 Induction Algorithm

Machine Learning
Pruning Decision Trees with Misclassification Costs

ECML '98 Proceedings of the 10th European Conference on Machine Learning
Class Probability Estimation and Cost-Sensitive Classification Decisions

ECML '02 Proceedings of the 13th European Conference on Machine Learning
Issues in Classifier Evaluation using Optimal Cost Curves

ICML '02 Proceedings of the Nineteenth International Conference on Machine Learning
Learning Decision Trees Using the Area Under the ROC Curve

ICML '02 Proceedings of the Nineteenth International Conference on Machine Learning
The Case against Accuracy Estimation for Comparing Induction Algorithms

ICML '98 Proceedings of the Fifteenth International Conference on Machine Learning
Bootstrap Methods for the Cost-Sensitive Evaluation of Classifiers

ICML '00 Proceedings of the Seventeenth International Conference on Machine Learning
Exploiting the Cost (In)sensitivity of Decision Tree Splitting Criteria

ICML '00 Proceedings of the Seventeenth International Conference on Machine Learning
Cost-Sensitive Specialization

PRICAI '96 Proceedings of the 4th Pacific Rim International Conference on Artificial Intelligence: Topics in Artificial Intelligence
Cost-Sensitive Learning by Cost-Proportionate Example Weighting

ICDM '03 Proceedings of the Third IEEE International Conference on Data Mining
On the Application of ROC Analysis to Predict Classification Performance Under Varying Class Distributions

Machine Learning
ROC confidence bands: an empirical evaluation

ICML '05 Proceedings of the 22nd international conference on Machine learning
Data Mining: Practical Machine Learning Tools and Techniques, Second Edition (Morgan Kaufmann Series in Data Management Systems)

Data Mining: Practical Machine Learning Tools and Techniques, Second Edition (Morgan Kaufmann Series in Data Management Systems)
Learning when training data are costly: the effect of class distribution on tree induction

Journal of Artificial Intelligence Research
Cost-sensitive classification: empirical evaluation of a hybrid genetic decision tree induction algorithm

Journal of Artificial Intelligence Research
A novelty detection approach to classification

IJCAI'95 Proceedings of the 14th international joint conference on Artificial intelligence - Volume 1
The use of the area under the ROC curve in the evaluation of machine learning algorithms

Pattern Recognition

Rule Extraction from Support Vector Machines: A Sequential Covering Approach

IEEE Transactions on Knowledge and Data Engineering
Instance weighting versus threshold adjusting for cost-sensitive classification

Knowledge and Information Systems
Pointwise exact bootstrap distributions of cost curves

Proceedings of the 25th international conference on Machine learning
Maximizing classifier utility when there are data acquisition and modeling costs

Data Mining and Knowledge Discovery
Automatically countering imbalance and its empirical relationship to cost

Data Mining and Knowledge Discovery
Techniques for evaluating fault prediction models

Empirical Software Engineering
A Cost-Sensitive Approach to Feature Selection in Micro-Array Data Classification

WILF '07 Proceedings of the 7th international workshop on Fuzzy Logic and Applications: Applications of Fuzzy Sets Theory
Classification and Quantification Based on Image Analysis for Sperm Samples with Uncertain Damaged/Intact Cell Proportions

ICIAR '08 Proceedings of the 5th international conference on Image Analysis and Recognition
A Projection-Based Framework for Classifier Performance Evaluation

ECML PKDD '08 Proceedings of the 2008 European Conference on Machine Learning and Knowledge Discovery in Databases - Part I
Quantifying the proportion of damaged sperm cells based on image analysis and neural networks

SMO'08 Proceedings of the 8th conference on Simulation, modelling and optimization
EnsembleMatrix: interactive visualization to support machine learning with multiple classifiers

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Information Market-Based Decision Fusion

Management Science
Misclassification cost-sensitive fault prediction models

PROMISE '09 Proceedings of the 5th International Conference on Predictor Models in Software Engineering
Domain driven data mining to improve promotional campaign ROI and select marketing channels

Proceedings of the 18th ACM conference on Information and knowledge management
Cost curve analysis of biometric system performance

BTAS'09 Proceedings of the 3rd IEEE international conference on Biometrics: Theory, applications and systems
Interactive optimization for steering machine classification

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
CSNL: A cost-sensitive non-linear decision tree algorithm

ACM Transactions on Knowledge Discovery from Data (TKDD)
Capturing heuristics and intelligent methods for improving micro-array data classification

IDEAL'07 Proceedings of the 8th international conference on Intelligent data engineering and automated learning
Spam filtering and email-mediated applications

WImBI'06 Proceedings of the 1st WICI international conference on Web intelligence meets brain informatics
Cost-sensitive classifier evaluation using cost curves

PAKDD'08 Proceedings of the 12th Pacific-Asia conference on Advances in knowledge discovery and data mining
Assessing the impact of changing environments on classifier performance

Canadian AI'08 Proceedings of the Canadian Society for computational studies of intelligence, 21st conference on Advances in artificial intelligence
Learning with cost intervals

Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining
Defect prediction using social network analysis on issue repositories

Proceedings of the 2011 International Conference on Software and Systems Process
Information, Divergence and Risk for Binary Experiments

The Journal of Machine Learning Research
Inferring multiple graphical structures

Statistics and Computing
An alternative to ROC and AUC analysis of classifiers

IDA'11 Proceedings of the 10th international conference on Advances in intelligent data analysis X
On the dataset shift problem in software engineering prediction models

Empirical Software Engineering
Robustness of classifiers to changing environments

AI'10 Proceedings of the 23rd Canadian conference on Advances in Artificial Intelligence
Discriminative vs. generative classifiers for cost sensitive learning

AI'06 Proceedings of the 19th international conference on Advances in Artificial Intelligence: Canadian Society for Computational Studies of Intelligence
Towards cost-sensitive learning for real-world applications

PAKDD'11 Proceedings of the 15th international conference on New Frontiers in Applied Data Mining
Class distribution estimation based on the Hellinger distance

Information Sciences: an International Journal
Design and Analysis of Classifier Learning Experiments in Bioinformatics: Survey and Case Studies

IEEE/ACM Transactions on Computational Biology and Bioinformatics (TCBB)
A comparison study of cost-sensitive classifier evaluations

BI'12 Proceedings of the 2012 international conference on Brain Informatics
ROC curve equivalence using the Kolmogorov-Smirnov test

Pattern Recognition Letters
Link Prediction: Fair and Effective Evaluation

ASONAM '12 Proceedings of the 2012 International Conference on Advances in Social Networks Analysis and Mining (ASONAM 2012)
Comparing multi-objective and threshold-moving ROC curve generation for a prototype-based classifier

Proceedings of the 15th annual conference on Genetic and evolutionary computation
A unified view of performance metrics: translating threshold choice into expected classification loss

The Journal of Machine Learning Research
Assessing scorecard performance: A literature review and classification

Expert Systems with Applications: An International Journal
ROC curves for regression

Pattern Recognition
Area under the distance threshold curve as an evaluation measure for probabilistic classifiers

MLDM'13 Proceedings of the 9th international conference on Machine Learning and Data Mining in Pattern Recognition
ROC curves in cost space

Machine Learning
Training and assessing classification rules with imbalanced data

Data Mining and Knowledge Discovery
Assessment of data quality in accounting data with association rules

Expert Systems with Applications: An International Journal
Half-AUC for the evaluation of sensitive or specific classifiers

Pattern Recognition Letters
Cost-sensitive three-way email spam filtering

Journal of Intelligent Information Systems
ROC analysis of classifiers in machine learning: A survey

Intelligent Data Analysis

Quantified Score

Hi-index	0.00

Visualization

Abstract

This paper introduces cost curves, a graphical technique for visualizing the performance (error rate or expected cost) of 2-class classifiers over the full range of possible class distributions and misclassification costs. Cost curves are shown to be superior to ROC curves for visualizing classifier performance for most purposes. This is because they visually support several crucial types of performance assessment that cannot be done easily with ROC curves, such as showing confidence intervals on a classifier's performance, and visualizing the statistical significance of the difference in performance of two classifiers. A software tool supporting all the cost curve analysis described in this paper is available from the authors.