The use of the area under the ROC curve in the evaluation of machine learning algorithms

Authors:
Andrew P. Bradley
Affiliations:
Cooperative Research Centre for Sensor Signal and Information Processing, Department of Electrical and Computer Engineering, The University of Queensland, QLD 4072, Australia
Venue:
Pattern Recognition
Year:
1997

Citing 7
Cited 397

Decision estimation and classification: an introduction to pattern recognition and related topics

Decision estimation and classification: an introduction to pattern recognition and related topics
Models of incremental concept formation

Artificial Intelligence
Introduction to statistical pattern recognition (2nd ed.)

Introduction to statistical pattern recognition (2nd ed.)
Learning internal representations by error propagation

Parallel distributed processing: explorations in the microstructure of cognition, vol. 1
C4.5: programs for machine learning

C4.5: programs for machine learning
Overfitting Avoidance as Bias

Machine Learning
The Multiscale Classifier

IEEE Transactions on Pattern Analysis and Machine Intelligence

Robust classification systems for imprecise environments

AAAI '98/IAAI '98 Proceedings of the fifteenth national/tenth conference on Artificial intelligence/Innovative applications of artificial intelligence
Robust Classification for Imprecise Environments

Machine Learning
A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems

Machine Learning
Class Probability Estimation and Cost-Sensitive Classification Decisions

ECML '02 Proceedings of the 13th European Conference on Machine Learning
An Optimal Reject Rule for Binary Classifiers

Proceedings of the Joint IAPR International Workshops on Advances in Pattern Recognition
An Automated ILP Server in the Field of Bioinformatics

ILP '01 Proceedings of the 11th International Conference on Inductive Logic Programming
Combining One-Class Classifiers

MCS '01 Proceedings of the Second International Workshop on Multiple Classifier Systems
Non-retrieval: Blocking Pornographic Images

CIVR '02 Proceedings of the International Conference on Image and Video Retrieval
Toward Bayesian Classifiers with Accurate Probabilities

PAKDD '02 Proceedings of the 6th Pacific-Asia Conference on Advances in Knowledge Discovery and Data Mining
A machine learning approach for the curation of biomedical literature: KDD Cup 2002 (task 1)

ACM SIGKDD Explorations Newsletter
Model-based detection, segmentation, and classification for image analysis using on-line shape learning

Machine Vision and Applications
Tree Induction for Probability-Based Ranking

Machine Learning
Improved Rooftop Detection in Aerial Images with Machine Learning

Machine Learning
Tree induction vs. logistic regression: a learning-curve analysis

The Journal of Machine Learning Research
Effectiveness of Information Extraction, Multi-Relational, and Semi-Supervised Learning for Predicting Functional Properties of Genes

ICDM '03 Proceedings of the Third IEEE International Conference on Data Mining
Comparing Naive Bayes, Decision Trees, and SVM with AUC and Accuracy

ICDM '03 Proceedings of the Third IEEE International Conference on Data Mining
Impact Studies and Sensitivity Analysis in Medical Data Mining with ROC-based Genetic Learning

ICDM '03 Proceedings of the Third IEEE International Conference on Data Mining
Aggregation-based feature invention and relational concept classes

Proceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining
Categorizing web queries according to geographical locality

CIKM '03 Proceedings of the twelfth international conference on Information and knowledge management
Support Vector Data Description

Machine Learning
A General Model for Finite-Sample Effects in Training and Testing of Competing Classifiers

IEEE Transactions on Pattern Analysis and Machine Intelligence
Active Sampling for Class Probability Estimation and Ranking

Machine Learning
Classifying biological articles using web resources

Proceedings of the 2004 ACM symposium on Applied computing
Applying inductive logic programming to predicting gene function

AI Magazine
Mining with rarity: a unifying framework

ACM SIGKDD Explorations Newsletter - Special issue on learning from imbalanced datasets
Evaluation of decision trees: a multi-criteria approach

Computers and Operations Research
Multi-Relational Learning, Text Mining, and Semi-Supervised Learning for Functional Genomics

Machine Learning
Co-EM support vector learning

ICML '04 Proceedings of the twenty-first international conference on Machine learning
Optimising area under the ROC curve using gradient descent

ICML '04 Proceedings of the twenty-first international conference on Machine learning
Shape-Based Recognition of Wiry Objects

IEEE Transactions on Pattern Analysis and Machine Intelligence
A critical review of multi-objective optimization in data mining: a position paper

ACM SIGKDD Explorations Newsletter
Using AUC and Accuracy in Evaluating Learning Algorithms

IEEE Transactions on Knowledge and Data Engineering
On the application of ROC analysis to predict classification performance under varying class distributions

Machine Learning
KBA: Kernel Boundary Alignment Considering Imbalanced Data Distribution

IEEE Transactions on Knowledge and Data Engineering
An experimental investigation of the impact of aggregation on the performance of data mining with logistic regression

Information and Management
Local sparsity control for naive Bayes with extreme misclassification costs

Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining
Predicting the product purchase patterns of corporate customers

Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining
Learning Yeast Gene Functions from Heterogeneous Sources of Data Using Hybrid Weighted Bayesian Networks

CSB '05 Proceedings of the 2005 IEEE Computational Systems Bioinformatics Conference
Adapting the CBA algorithm by means of intensity of implication

Information Sciences—Informatics and Computer Science: An International Journal - Special issue: Dealing with uncertainty in data mining and information extraction
Case studies in the use of ROC curve analysis for sensor-based estimates in human computer interaction

GI '05 Proceedings of Graphics Interface 2005
Utility based data mining for time series analysis: cost-sensitive learning for neural network predictors

UBDM '05 Proceedings of the 1st international workshop on Utility-based data mining
Gene classification: issues and challenges for relational learning

MRDM '05 Proceedings of the 4th international workshop on Multi-relational mining
A ROC-based reject rule for dichotomizers

Pattern Recognition Letters
ROC confidence bands: an empirical evaluation

ICML '05 Proceedings of the 22nd international conference on Machine learning
Supervised versus multiple instance learning: an empirical comparison

ICML '05 Proceedings of the 22nd international conference on Machine learning
Augmenting naive Bayes for ranking

ICML '05 Proceedings of the 22nd international conference on Machine learning
Learning Instance Greedily Cloning Naive Bayes for Ranking

ICDM '05 Proceedings of the Fifth IEEE International Conference on Data Mining
Ranking-Based Evaluation of Regression Models

ICDM '05 Proceedings of the Fifth IEEE International Conference on Data Mining
Partial Ensemble Classifiers Selection for Better Ranking

ICDM '05 Proceedings of the Fifth IEEE International Conference on Data Mining
An Assessment of Case-Based Reasoning for Spam Filtering

Artificial Intelligence Review
Distribution-based aggregation for relational learning with identifier attributes

Machine Learning
Improving the Practice of Classifier Performance Assessment

Neural Computation
Estimating the uncertainty in the estimated mean area under the ROC curve of a classifier

Pattern Recognition Letters
Bayesian Segmental Models with Multiple Sequence Alignment Profiles for Protein Secondary Structure and Contact Map Prediction

IEEE/ACM Transactions on Computational Biology and Bioinformatics (TCBB)
A methodology for comparing classifiers that allow the control of bias

Proceedings of the 2006 ACM symposium on Applied computing
Evaluating the performance of cost-based discretization versus entropy-and error-based discretization

Computers and Operations Research
The relationship between Precision-Recall and ROC curves

ICML '06 Proceedings of the 23rd international conference on Machine learning
Gleaner: Creating ensembles of first-order clauses to improve recall-precision curves

Machine Learning
Preface: Special issue on "ROC Analysis in Pattern Recognition"

Pattern Recognition Letters - Special issue: ROC analysis in pattern recognition
An introduction to ROC analysis

Pattern Recognition Letters - Special issue: ROC analysis in pattern recognition
ROC curves and video analysis optimization in intestinal capsule endoscopy

Pattern Recognition Letters - Special issue: ROC analysis in pattern recognition
Exploiting AUC for optimal linear combinations of dichotomizers

Pattern Recognition Letters - Special issue: ROC analysis in pattern recognition
The interaction between classification and reject performance for distance-based reject-option classifiers

Pattern Recognition Letters - Special issue: ROC analysis in pattern recognition
Multi-class ROC analysis from a multi-objective optimisation perspective

Pattern Recognition Letters - Special issue: ROC analysis in pattern recognition
Application of LVQ to novelty detection using outlier training data

Pattern Recognition Letters
Cost curves: An improved method for visualizing classifier performance

Machine Learning
Quantitative characterization and prediction of on-line purchasing behavior: a latent variable approach

International Journal of Electronic Commerce - Special issue: Electronic intermediaries and networks in business-to-business electronic commerce
Classifier evaluation under limited resources

Pattern Recognition Letters
The effect of imbalanced data sets on LDA: A theoretical and empirical analysis

Pattern Recognition
Selecting features in microarray classification using ROC curves

Pattern Recognition
Post-pruning in decision tree induction using multiple performance measures

Computers and Operations Research
The feasibility of constructing a Predictive Outcome Model for breast cancer using the tools of data mining

Expert Systems with Applications: An International Journal
Rule Extraction from Support Vector Machines: A Sequential Covering Approach

IEEE Transactions on Knowledge and Data Engineering
On the Dimensionality of Face Space

IEEE Transactions on Pattern Analysis and Machine Intelligence
Comparison of anomaly signal quality in common detection metrics

Proceedings of the 3rd annual ACM workshop on Mining network data
Predicting Metastasis in Breast Cancer: Comparing a Decision Tree with Domain Experts

Journal of Medical Systems
Diagnosing scrapie in sheep: A classification experiment

Computers in Biology and Medicine
From outliers to prototypes: Ordering data

Neurocomputing
Evaluating and Tuning Predictive Data Mining Models Using Receiver Operating Characteristic Curves

Journal of Management Information Systems
Approximating the multiclass ROC by pairwise analysis

Pattern Recognition Letters
Ranking-based evaluation of regression models

Knowledge and Information Systems
An integrated statistical model for multimedia evidence combination

Proceedings of the 15th international conference on Multimedia
A surface-based approach for classification of 3D neuroanatomic structures

Intelligent Data Analysis
Email answering assistance by semi-supervised text classification

Intelligent Data Analysis
Ensemble methods for anomaly detection and distributed intrusion detection in Mobile Ad-Hoc Networks

Information Fusion
Development of Two-Stage SVM-RFE Gene Selection Strategy for Microarray Expression Data Analysis

IEEE/ACM Transactions on Computational Biology and Bioinformatics (TCBB)
Post-pruning in regression tree induction: An integrated approach

Expert Systems with Applications: An International Journal
Do unbalanced data have a negative effect on LDA?

Pattern Recognition
Do unbalanced data have a negative effect on LDA?

Pattern Recognition
Maximizing the area under the ROC curve by pairwise feature combination

Pattern Recognition
2008 Special Issue: Training neural network classifiers for medical decision making: The effects of imbalanced datasets on classification performance

Neural Networks
RISP: A web-based server for prediction of RNA-binding sites in proteins

Computer Methods and Programs in Biomedicine
Detecting worm variants using machine learning

CoNEXT '07 Proceedings of the 2007 ACM CoNEXT conference
Improving accuracy in astrocytomas grading by integrating a robust least squares mapping driven support vector machine classifier into a two level grade classification scheme

Computer Methods and Programs in Biomedicine
AdaBoost with SVM-based component classifiers

Engineering Applications of Artificial Intelligence
Instance weighting versus threshold adjusting for cost-sensitive classification

Knowledge and Information Systems
A boosting algorithm for learning bipartite ranking functions with partially labeled data

Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval
A critical analysis of variants of the AUC

Machine Learning
PRIE: a system for generating rulelists to maximize ROC performance

Data Mining and Knowledge Discovery
Utility of multilayer perceptron neural network classifiers in the diagnosis of the obstructive sleep apnoea syndrome from nocturnal oximetry

Computer Methods and Programs in Biomedicine
A boundary method for outlier detection based on support vector domain description

Pattern Recognition
Efficient AUC Optimization for Classification

PKDD 2007 Proceedings of the 11th European conference on Principles and Practice of Knowledge Discovery in Databases
A Fault Prediction Model with Limited Fault Data to Improve Test Process

PROFES '08 Proceedings of the 9th international conference on Product-Focused Software Process Improvement
Improving k-Nearest Neighbour Classification with Distance Functions Based on Receiver Operating Characteristics

ECML PKDD '08 Proceedings of the 2008 European Conference on Machine Learning and Knowledge Discovery in Databases - Part I
Smooth Boosting for Margin-Based Ranking

ALT '08 Proceedings of the 19th international conference on Algorithmic Learning Theory
Evaluation and Comparison of Inferred Regular Grammars

ICGI '08 Proceedings of the 9th international colloquium on Grammatical Inference: Algorithms and Applications
Naive Bayes for optimal ranking

Journal of Experimental & Theoretical Artificial Intelligence
Online generation of scene descriptions in urban environments

Robotics and Autonomous Systems
The risk-utility tradeoff for IP address truncation

Proceedings of the 1st ACM workshop on Network data anonymization
Incorporating domain knowledge into data mining classifiers: An application in indirect lending

Decision Support Systems
Boosting and measuring the performance of ensembles for a successful database marketing

Expert Systems with Applications: An International Journal
An iterative multi-scale tensor voting scheme for perceptual grouping of natural shapes in cluttered backgrounds

Computer Vision and Image Understanding
Handling imbalanced data sets with a modification of Decorate algorithm

International Journal of Computer Applications in Technology
An evaluation of one-class classification techniques for speaker verification

Artificial Intelligence Review
Learning Curves for the Analysis of Multiple Instance Classifiers

SSPR & SPR '08 Proceedings of the 2008 Joint IAPR International Workshop on Structural, Syntactic, and Statistical Pattern Recognition
Combination of Experts by Classifiers in Similarity Score Spaces

SSPR & SPR '08 Proceedings of the 2008 Joint IAPR International Workshop on Structural, Syntactic, and Statistical Pattern Recognition
A Combined Classification Algorithm Based on C4.5 and NB

ISICA '08 Proceedings of the 3rd International Symposium on Advances in Computation and Intelligence
Multi-Relational Classification in Imbalanced Domains

ISICA '08 Proceedings of the 3rd International Symposium on Advances in Computation and Intelligence
Minimum spanning tree based one-class classifier

Neurocomputing
McPAD: A multiple classifier system for accurate payload-based anomaly detection

Computer Networks: The International Journal of Computer and Telecommunications Networking
An efficient algorithm for learning to rank from preference graphs

Machine Learning
Using enhanced genetic programming techniques for evolving classifiers in the context of medical diagnosis

Genetic Programming and Evolvable Machines
Dynamic Exponential Family Matrix Factorization

PAKDD '09 Proceedings of the 13th Pacific-Asia Conference on Advances in Knowledge Discovery and Data Mining
Safe-Level-SMOTE: Safe-Level-Synthetic Minority Over-Sampling TEchnique for Handling the Class Imbalanced Problem

PAKDD '09 Proceedings of the 13th Pacific-Asia Conference on Advances in Knowledge Discovery and Data Mining
The Time-Series Link Prediction Problem with Applications in Communication Surveillance

INFORMS Journal on Computing
Genre-based decomposition of email class noise

Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining
Set Cover Feature Selection for Text Categorisation and spam detection

International Journal of Advanced Intelligence Paradigms
Efficient AUC Maximization with Regularized Least-Squares

Proceedings of the 2008 conference on Tenth Scandinavian Conference on Artificial Intelligence: SCAI 2008
A graph kernel for protein-protein interaction extraction

BioNLP '08 Proceedings of the Workshop on Current Trends in Biomedical Natural Language Processing
Index of Balanced Accuracy: A Performance Measure for Skewed Class Distributions

IbPRIA '09 Proceedings of the 4th Iberian Conference on Pattern Recognition and Image Analysis
Terrain Segmentation with On-Line Mixtures of Experts for Autonomous Robot Navigation

MCS '09 Proceedings of the 8th International Workshop on Multiple Classifier Systems
Incremental Kernel Machines for Protein Remote Homology Detection

HAIS '09 Proceedings of the 4th International Conference on Hybrid Artificial Intelligence Systems
A computer-aided system for malignancy risk assessment of nodules in thyroid US images based on boundary features

Computer Methods and Programs in Biomedicine
Dynamic Score Combination: A Supervised and Unsupervised Score Combination Method

MLDM '09 Proceedings of the 6th International Conference on Machine Learning and Data Mining in Pattern Recognition
Aligning Bayesian Network Classifiers with Medical Contexts

MLDM '09 Proceedings of the 6th International Conference on Machine Learning and Data Mining in Pattern Recognition
Measuring classifier performance: a coherent alternative to the area under the ROC curve

Machine Learning
SMOTE: synthetic minority over-sampling technique

Journal of Artificial Intelligence Research
Learning when training data are costly: the effect of class distribution on tree induction

Journal of Artificial Intelligence Research
Learning from labeled and unlabeled data: an empirical study across techniques and domains

Journal of Artificial Intelligence Research
Gesture salience as a hidden variable for coreference resolution and keyframe extraction

Journal of Artificial Intelligence Research
Keep the decision tree and estimate the class probabilities using its decision boundary

IJCAI'07 Proceedings of the 20th international joint conference on Artifical intelligence
Constructing new and better evaluation measures for machine learning

IJCAI'07 Proceedings of the 20th international joint conference on Artifical intelligence
AUC: a statistically consistent and more discriminating measure than accuracy

IJCAI'03 Proceedings of the 18th international joint conference on Artificial intelligence
Bayesian Order-Consistency Testing with Class Priors Derivation for Robust Change Detection

AVSS '09 Proceedings of the 2009 Sixth IEEE International Conference on Advanced Video and Signal Based Surveillance
Evolutionary undersampling for classification with imbalanced datasets: Proposals and taxonomy

Evolutionary Computation
Active learning for class probability estimation and ranking

IJCAI'01 Proceedings of the 17th international joint conference on Artificial intelligence - Volume 2
On the use of classification reliability for improving performance of the one-per-class decomposition method

Data & Knowledge Engineering
SVMs modeling for highly imbalanced classification

IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics - Special issue on human computing
Exploratory undersampling for class-imbalance learning

IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics
Dimensionality reduction for density ratio estimation in high-dimensional spaces

Neural Networks
A multi-model selection framework for unknown and/or evolutive misclassification cost problems

Pattern Recognition
Abnormal activity recognition based on HDP-HMM models

IJCAI'09 Proceedings of the 21st international jont conference on Artifical intelligence
Differentiating between individual class performance in genetic programming fitness for classification with unbalanced data

CEC'09 Proceedings of the Eleventh conference on Congress on Evolutionary Computation
Classification of Imbalanced Data Sets by Using the Hybrid Re-sampling Algorithm Based on Isomap

ISICA '09 Proceedings of the 4th International Symposium on Advances in Computation and Intelligence
Multi-Objective Genetic Programming for Classification with Unbalanced Data

AI '09 Proceedings of the 22nd Australasian Joint Conference on Advances in Artificial Intelligence
A novel measure for evaluating classifiers

Expert Systems with Applications: An International Journal
Evaluating classifiers: relation between area under the receiver operator characteristic curve and overall accuracy

IJCNN'09 Proceedings of the 2009 international joint conference on Neural Networks
The effect of class imbalance on case selection for case-based classifiers, with emphasis on computer-aided diagnosis systems

IJCNN'09 Proceedings of the 2009 international joint conference on Neural Networks
An experimental investigation of the impact of aggregation on the performance of data mining with logistic regression

Information and Management
Adapting the CBA algorithm by means of intensity of implication

Information Sciences: an International Journal
BioPPISVMExtractor: A protein-protein interaction extractor for biomedical literature using SVM and rich feature sets

Journal of Biomedical Informatics
On the 2-tuples based genetic tuning performance for fuzzy rule based classification systems in imbalanced data-sets

Information Sciences: an International Journal
Formal verification of wastewater treatment processes using events detected from continuous signals by means of artificial neural networks. Case study: SBR plant

Environmental Modelling & Software
Sparse Support Vector Machines with L_{p} Penalty for Biomarker Identification

IEEE/ACM Transactions on Computational Biology and Bioinformatics (TCBB)
Why fuzzy decision trees are good rankers

IEEE Transactions on Fuzzy Systems
Classification of Breast Cancer Malignancy Using Cytological Images of Fine Needle Aspiration Biopsies

International Journal of Applied Mathematics and Computer Science - Applied Image Processing
A framework for case-based diagnosis of batch processes in the principal components space

ETFA'09 Proceedings of the 14th IEEE international conference on Emerging technologies & factory automation
On the ability of complexity metrics to predict fault-prone classes in object-oriented systems

Journal of Systems and Software
A symbolic fault-prediction model based on multiobjective particle swarm optimization

Journal of Systems and Software
AUC maximization linear classifier based on active learning and its application

Neurocomputing
A Least-squares Approach to Direct Importance Estimation

The Journal of Machine Learning Research
A machine learning approach for the curation of biomedical literature

ECIR'03 Proceedings of the 25th European conference on IR research
A ROC-based reject rule for support vector machines

MLDM'03 Proceedings of the 3rd international conference on Machine learning and data mining in pattern recognition
AUC: a better measure than accuracy in comparing learning algorithms

AI'03 Proceedings of the 16th Canadian society for computational studies of intelligence conference on Advances in artificial intelligence
Disease modeling using evolved discriminate function

EuroGP'03 Proceedings of the 6th European conference on Genetic programming
An empirical boosting scheme for ROC-based genetic programming classifiers

EuroGP'07 Proceedings of the 10th European conference on Genetic programming
Combining supervised and semi-supervised classifier for personalized spam filtering

PAKDD'07 Proceedings of the 11th Pacific-Asia conference on Advances in knowledge discovery and data mining
The ROC skeleton for multiclass ROC estimation

Pattern Recognition Letters
A sorting optimization curve with quality and yield requirements

Pattern Recognition Letters
Making class bias useful: a strategy of learning from imbalanced data

IDEAL'07 Proceedings of the 8th international conference on Intelligent data engineering and automated learning
Learning locally weighted C4.4 for class probability estimation

DS'07 Proceedings of the 10th international conference on Discovery science
Fitness functions in genetic programming for classification with unbalanced data

AI'07 Proceedings of the 20th Australian joint conference on Advances in artificial intelligence
A framework for modeling positive class expansion with single snapshot

PAKDD'08 Proceedings of the 12th Pacific-Asia conference on Advances in knowledge discovery and data mining
Iterative Boolean combination of classifiers in the ROC space: An application to anomaly detection with HMMs

Pattern Recognition
Comparison of pleomorphic and structural features used for breast cancer malignancy classification

Canadian AI'08 Proceedings of the Canadian Society for computational studies of intelligence, 21st conference on Advances in artificial intelligence
GP classification under imbalanced data sets: active sub-sampling and AUC approximation

EuroGP'08 Proceedings of the 11th European conference on Genetic programming
Derivations of normalized mutual information in binary classifications

FSKD'09 Proceedings of the 6th international conference on Fuzzy systems and knowledge discovery - Volume 1
Putting it all together: using socio-technical networks to predict failures

ISSRE'09 Proceedings of the 20th IEEE international conference on software reliability engineering
When to choose an ensemble classifier model for data mining

International Journal of Business Intelligence and Data Mining
An ensemble-based evolutionary framework for coping with distributed intrusion detection

Genetic Programming and Evolvable Machines
Realism assessment of color compatibility using a single image

ICIP'09 Proceedings of the 16th IEEE international conference on Image processing
Nearest neighbour group-based classification

Pattern Recognition
Evaluating logistic regression models to estimate software project outcomes

Information and Software Technology
Toward comparison-based adaptive operator selection

Proceedings of the 12th annual conference on Genetic and evolutionary computation
AUC analysis of the pareto-front using multi-objective GP for classification with unbalanced data

Proceedings of the 12th annual conference on Genetic and evolutionary computation
Genetic rule extraction optimizing brier score

Proceedings of the 12th annual conference on Genetic and evolutionary computation
Fitness-AUC bandit adaptive strategy selection vs. the probability matching one within differential evolution: an empirical comparison on the bbob-2010 noiseless testbed

Proceedings of the 12th annual conference companion on Genetic and evolutionary computation
An intervention mechanism for assistive living in smart homes

Journal of Ambient Intelligence and Smart Environments
Learning with cost intervals

Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining
Combined regression and ranking

Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining
Two information-theoretic tools to assess the performance of multi-class classifiers

Pattern Recognition Letters
Multistage Gene Normalization and SVM-Based Ranking for Protein Interactor Extraction in Full-Text Articles

IEEE/ACM Transactions on Computational Biology and Bioinformatics (TCBB)
Random forest-based prediction of protein sumoylation sites from sequence features

Proceedings of the First ACM International Conference on Bioinformatics and Computational Biology
Memetic Pareto Evolutionary Artificial Neural Networks to determine growth/no-growth in predictive microbiology

Applied Soft Computing
Language independent system for definition extraction: first results using learning algorithms

WDE '09 Proceedings of the 1st Workshop on Definition Extraction
Analysis of early late phase in single-and dual-frequency GPS receivers for multipath detection

GPS Solutions
Neural Network Classifier with Entropy Based Feature Selection on Breast Cancer Diagnosis

Journal of Medical Systems
Functional identification of biological neural networks using reservoir adaptation for point processes

Journal of Computational Neuroscience
A state-space approach to optimal level-crossing prediction for linear Gaussian processes

IEEE Transactions on Information Theory
An Analysis of the Impact of Passenger Profiling for Transportation Security

Operations Research
Tree induction over perennial objects

SSDBM'10 Proceedings of the 22nd international conference on Scientific and statistical database management
Local PCA regression for missing data estimation in telecommunication dataset

PRICAI'10 Proceedings of the 11th Pacific Rim international conference on Trends in artificial intelligence
Comparison-based adaptive strategy selection with bandits in differential evolution

PPSN'10 Proceedings of the 11th international conference on Parallel problem solving from nature: Part I
Feature selection for multi-purpose predictive models: a many-objective task

PPSN'10 Proceedings of the 11th international conference on Parallel problem solving from nature: Part I
Using evolutionary multiobjective techniques for imbalanced classification data

ICANN'10 Proceedings of the 20th international conference on Artificial neural networks: Part I
A hybrid approach for artifact detection in EEG data

ICANN'10 Proceedings of the 20th international conference on Artificial neural networks: Part I
Combination of dichotomizers for maximizing the partial area under the ROC curve

SSPR&SPR'10 Proceedings of the 2010 joint IAPR international conference on Structural, syntactic, and statistical pattern recognition
Rule extraction from support vector machines: A review

Neurocomputing
Towards the Generic Framework for Utility Considerations in Data Mining Research

Proceedings of the 2010 conference on Data Mining for Business Applications
Abnormality detection for improving elder's daily life independent

ICOST'10 Proceedings of the Aging friendly technology for health and independence, and 8th international conference on Smart homes and health telematics
Learning classifiers from imbalanced data based on biased minimax probability machine

CVPR'04 Proceedings of the 2004 IEEE computer society conference on Computer vision and pattern recognition
Advances in applying genetic programming to machine learning, focussing on classification problems

IPDPS'06 Proceedings of the 20th international conference on Parallel and distributed processing
Non-linear parametric Bayesian regression for robust background subtraction

WMVC'09 Proceedings of the 2009 international conference on Motion and video computing
Determining the optimal re-sampling strategy for a classification model with imbalanced data using design of experiments and response surface methodologies

Expert Systems with Applications: An International Journal
Random one-dependence estimators

Pattern Recognition Letters
An experimental comparison of cross-validation techniques for estimating the area under the ROC curve

Computational Statistics & Data Analysis
An intraday market risk management approach based on textual analysis

Decision Support Systems
Direct density-ratio estimation with dimensionality reduction via least-squares hetero-distributional subspace search

Neural Networks
A preliminary study on the selection of generalized instances for imbalanced classification

IEA/AIE'10 Proceedings of the 23rd international conference on Industrial engineering and other applications of applied intelligent systems - Volume Part I
Using genetic K-means algorithm for PCA regression data in customer churn prediction

ADMA'10 Proceedings of the 6th international conference on Advanced data mining and applications - Volume Part II
Using PCA to predict customer churn in telecommunication dataset

ADMA'10 Proceedings of the 6th international conference on Advanced data mining and applications - Volume Part II
Hierarchical classification with dynamic-threshold SVM ensemble for gene function prediction

ADMA'10 Proceedings of the 6th international conference on Advanced data mining and applications - Volume Part II
Why Does Collaborative Filtering Work? Transaction-Based Recommendation Model Validation and Selection by Analyzing Bipartite Random Graphs

INFORMS Journal on Computing
A new weighted approach to imbalanced data classification problem via support vector machine with quadratic cost function

Expert Systems with Applications: An International Journal
Breast cancer classification applying artificial metaplasticity algorithm

Neurocomputing
Cost-sensitive case-based reasoning using a genetic algorithm: Application to medical diagnosis

Artificial Intelligence in Medicine
Nonlinear dimensionality reduction for efficient and effective audio similarity searching

Multimedia Tools and Applications
Learning random forests for ranking

Frontiers of Computer Science in China
WBCD breast cancer database classification applying artificial metaplasticity neural network

Expert Systems with Applications: An International Journal
Multiple kernel learning in protein-protein interaction extraction from biomedical literature

Artificial Intelligence in Medicine
Recommendation for English multiple-choice cloze questions based on expected test scores

International Journal of Knowledge-based and Intelligent Engineering Systems
Using wavelet transform and multi-class least square support vector machine in multi-spectral imaging classification of Chinese famous tea

Expert Systems with Applications: An International Journal
Linguistic cost-sensitive learning of genetic fuzzy classifiers for imprecise data

International Journal of Approximate Reasoning
Training linear ranking SVMs in linearithmic time using red-black trees

Pattern Recognition Letters
Area under the ROC curve by bubble-sort approach (BSA)

ACMOS'05 Proceedings of the 7th WSEAS international conference on Automatic control, modeling and simulation
Learning of neural networks for fraud detection based on a partial area under curve

ISNN'05 Proceedings of the Second international conference on Advances in neural networks - Volume Part II
Robust Feature Selection for Microarray Data Based on Multicriterion Fusion

IEEE/ACM Transactions on Computational Biology and Bioinformatics (TCBB)
Software defect detection with rocus

Journal of Computer Science and Technology
Partial AUC maximization in a linear combination of dichotomizers

Pattern Recognition
Selecting training points for one-class support vector machines

Pattern Recognition Letters
Evaluating the change of software fault behavior with dataset attributes based on categorical correlation

Advances in Engineering Software
A rule-based method for customer churn prediction in telecommunication services

PAKDD'11 Proceedings of the 15th Pacific-Asia conference on Advances in knowledge discovery and data mining - Volume Part I
Data quality assurance and performance measurement of data mining for preventive maintenance of power grid

Proceedings of the First International Workshop on Data Mining for Service and Maintenance
User reputation in a comment rating environment

Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining
Complex wavelet transform variants in a scale invariant classification of celiac disease

IbPRIA'11 Proceedings of the 5th Iberian conference on Pattern recognition and image analysis
Addressing the classification with imbalanced data: open problems and new challenges on class distribution

HAIS'11 Proceedings of the 6th international conference on Hybrid artificial intelligent systems - Volume Part I
Margin-based over-sampling method for learning from imbalanced datasets

PAKDD'11 Proceedings of the 15th Pacific-Asia conference on Advances in knowledge discovery and data mining - Volume Part II
Improving k nearest neighbor with exemplar generalization for imbalanced classification

PAKDD'11 Proceedings of the 15th Pacific-Asia conference on Advances in knowledge discovery and data mining - Volume Part II
On the effectiveness of preprocessing methods when dealing with different levels of class imbalance

Knowledge-Based Systems
Evolutionary-based selection of generalized instances for imbalanced classification

Knowledge-Based Systems
iBAT: detecting anomalous taxi trajectories from GPS traces

Proceedings of the 13th international conference on Ubiquitous computing
A machine learning and data mining framework to enable evolutionary improvement in trauma triage

MLDM'11 Proceedings of the 7th international conference on Machine learning and data mining in pattern recognition
Customer churn prediction in telecommunications

Expert Systems with Applications: An International Journal
Comparing alternative classifiers for database marketing: The case of imbalanced datasets

Expert Systems with Applications: An International Journal
A novel ensemble algorithm for biomedical classification based on Ant Colony Optimization

Applied Soft Computing
Second-order polynomial models for background subtraction

ACCV'10 Proceedings of the 2010 international conference on Computer vision - Volume Part I
Anomaly detection using ensembles

MCS'11 Proceedings of the 10th international conference on Multiple classifier systems
Pruned random subspace method for one-class classifiers

MCS'11 Proceedings of the 10th international conference on Multiple classifier systems
Selection strategies for pAUC-based combination of dichotomizers

MCS'11 Proceedings of the 10th international conference on Multiple classifier systems
Tuning expert systems for cost-sensitive decisions

Advances in Artificial Intelligence
Dynamic classifier ensemble model for customer classification with imbalanced class distribution

Expert Systems with Applications: An International Journal
Processing and analysis of serum antibody binding signals from Printed Glycan Arrays for diagnostic and prognostic applications

International Journal of Bioinformatics Research and Applications
Processing and analysis of serum antibody binding signals from Printed Glycan Arrays for diagnostic and prognostic applications

International Journal of Bioinformatics Research and Applications
The effect of class imbalance on case selection for case-based classifiers: An empirical study in the context of medical decision support

Neural Networks
GP ensemble for distributed intrusion detection systems

ICAPR'05 Proceedings of the Third international conference on Advances in Pattern Recognition - Volume Part I
Combining accuracy and prior sensitivity for classifier design under prior uncertainty

SSPR'06/SPR'06 Proceedings of the 2006 joint IAPR international conference on Structural, Syntactic, and Statistical Pattern Recognition
Outlier detection using ball descriptions with adjustable metric

SSPR'06/SPR'06 Proceedings of the 2006 joint IAPR international conference on Structural, Syntactic, and Statistical Pattern Recognition
Improving the ranking performance of decision trees

ECML'06 Proceedings of the 17th European conference on Machine Learning
Experiments with SVM and stratified sampling with an imbalanced problem: detection of intestinal contractions

ICAPR'05 Proceedings of the Third international conference on Pattern Recognition and Image Analysis - Volume Part II
Many are better than one: improving probabilistic estimates from decision trees

MLCW'05 Proceedings of the First international conference on Machine Learning Challenges: evaluating Predictive Uncertainty Visual Object Classification, and Recognizing Textual Entailment
Rank measures for ordering

PKDD'05 Proceedings of the 9th European conference on Principles and Practice of Knowledge Discovery in Databases
Dynamic ensemble re-construction for better ranking

PKDD'05 Proceedings of the 9th European conference on Principles and Practice of Knowledge Discovery in Databases
Evolving neural networks for the classification of malignancy associated changes

IDEAL'05 Proceedings of the 6th international conference on Intelligent Data Engineering and Automated Learning
A new Fruit Fly Optimization Algorithm: Taking the financial distress model as an example

Knowledge-Based Systems
Improving Tree augmented Naive Bayes for class probability estimation

Knowledge-Based Systems
Cost-conscious comparison of supervised learning algorithms over multiple data sets

Pattern Recognition
Three-way analysis of structural health monitoring data

Neurocomputing
Robust SVM-based biomarker selection with noisy mass spectrometric proteomic data

EuroGP'06 Proceedings of the 2006 international conference on Applications of Evolutionary Computing
When will it happen?: relationship prediction in heterogeneous information networks

Proceedings of the fifth ACM international conference on Web search and data mining
Cost-sensitive ensemble of support vector machines for effective detection of microcalcification in breast cancer diagnosis

FSKD'05 Proceedings of the Second international conference on Fuzzy Systems and Knowledge Discovery - Volume Part II
Optimising two-stage recognition systems

MCS'05 Proceedings of the 6th international conference on Multiple Classifier Systems
Calculation of a composite DET curve

AVBPA'05 Proceedings of the 5th international conference on Audio- and Video-Based Biometric Person Authentication
Modeling individual and collaborative problem solving in medical problem-based learning

UM'05 Proceedings of the 10th international conference on User Modeling
One dependence augmented naive bayes

ADMA'05 Proceedings of the First international conference on Advanced Data Mining and Applications
Estimating the ROC curve of linearly combined dichotomizers

ICIAP'05 Proceedings of the 13th international conference on Image Analysis and Processing
Cyclic pattern kernels revisited

PAKDD'05 Proceedings of the 9th Pacific-Asia conference on Advances in Knowledge Discovery and Data Mining
Borderline-SMOTE: a new over-sampling method in imbalanced data sets learning

ICIC'05 Proceedings of the 2005 international conference on Advances in Intelligent Computing - Volume Part I
An online AUC formulation for binary classification

Pattern Recognition
Analysis of preprocessing vs. cost-sensitive learning for imbalanced classification. Open problems on intrinsic data characteristics

Expert Systems with Applications: An International Journal
The effect of attribute scaling on the performance of support vector machines

AI'04 Proceedings of the 17th Australian joint conference on Advances in Artificial Intelligence
Contributions of domain knowledge and stacked generalization in AI-Based classification models

AI'04 Proceedings of the 17th Australian joint conference on Advances in Artificial Intelligence
An effective support vector data description with relevant metric learning

ISNN'10 Proceedings of the 7th international conference on Advances in Neural Networks - Volume Part II
Uncertainty estimation with a finite dataset in the assessment of classification models

Computational Statistics & Data Analysis
Learning tree augmented naive bayes for ranking

DASFAA'05 Proceedings of the 10th international conference on Database Systems for Advanced Applications
The use of genetic programming for the construction of a financial management model in an enterprise

Applied Intelligence
Classifier variability: Accounting for training and testing

Pattern Recognition
Novelty detection in projected spaces for structural health monitoring

IDA'10 Proceedings of the 9th international conference on Advances in Intelligent Data Analysis
Discrimination-Based criteria for the evaluation of classifiers

FQAS'06 Proceedings of the 7th international conference on Flexible Query Answering Systems
Training classifiers for unbalanced distribution and cost-sensitive domains with ROC analysis

PKAW'06 Proceedings of the 9th Pacific Rim Knowledge Acquisition international conference on Advances in Knowledge Acquisition and Management
Feature weighted minimum distance classifier with multi-class confidence estimation

AI'06 Proceedings of the 19th Australian joint conference on Artificial Intelligence: advances in Artificial Intelligence
Weighted bagging for graph based one-class classifiers

MCS'10 Proceedings of the 9th international conference on Multiple Classifier Systems
Genetic programming for classification with unbalanced data

EuroGP'10 Proceedings of the 13th European conference on Genetic Programming
Segmentation-Driven recognition applied to numerical field extraction from handwritten incoming mail documents

DAS'06 Proceedings of the 7th international conference on Document Analysis Systems
An empirical study of bagging predictors for imbalanced data with different levels of class distribution

AI'11 Proceedings of the 24th international conference on Advances in Artificial Intelligence
Mixed-sampling approach to unbalanced data distributions: a case study involving Leukemia's document profiling

WSEAS Transactions on Information Science and Applications
Decision support for the software product line domain engineering lifecycle

Automated Software Engineering
Usage of Case-Based Reasoning, Neural Network and Adaptive Neuro-Fuzzy Inference System Classification Techniques in Breast Cancer Dataset Classification Diagnosis

Journal of Medical Systems
DBSMOTE: Density-Based Synthetic Minority Over-sampling TEchnique

Applied Intelligence
Not so greedy: Randomly Selected Naive Bayes

Expert Systems with Applications: An International Journal
Detection of Outlier Residues for Improving Interface Prediction in Protein Heterocomplexes

IEEE/ACM Transactions on Computational Biology and Bioinformatics (TCBB)
Hash Subgraph Pairwise Kernel for Protein-Protein Interaction Extraction

IEEE/ACM Transactions on Computational Biology and Bioinformatics (TCBB)
Applying instance-based techniques to prediction of final outcome in acute stroke

Artificial Intelligence in Medicine
Automatic identification of persian light verb constructions

CICLing'12 Proceedings of the 13th international conference on Computational Linguistics and Intelligent Text Processing - Volume Part I
AnyOut: anytime outlier detection on streaming data

DASFAA'12 Proceedings of the 17th international conference on Database Systems for Advanced Applications - Volume Part I
The AUK: A simple alternative to the AUC

Engineering Applications of Artificial Intelligence
A new search engine integrating hierarchical browsing and keyword search

IJCAI'11 Proceedings of the Twenty-Second international joint conference on Artificial Intelligence - Volume Volume Three
A combined SMOTE and PSO based RBF classifier for two-class imbalanced problems

Neurocomputing
Acute leukemia classification by ensemble particle swarm model selection

Artificial Intelligence in Medicine
Combining relevancy and methodological quality into a single ranking for evidence-based medicine

Information Sciences: an International Journal
SRF: a framework for the study of classifier behavior under training set mislabeling noise

PAKDD'12 Proceedings of the 16th Pacific-Asia conference on Advances in Knowledge Discovery and Data Mining - Volume Part I
A pruning-based approach for searching precise and generalized region for synthetic minority over-sampling

PAKDD'12 Proceedings of the 16th Pacific-Asia conference on Advances in Knowledge Discovery and Data Mining - Volume Part II
Computational intelligence for microarray data and biomedical image analysis for the early diagnosis of breast cancer

Expert Systems with Applications: An International Journal
A layered classification for malicious function identification and malware detection

Concurrency and Computation: Practice & Experience
Pedicle detection in planar radiographs based on image descriptors

ICIAR'12 Proceedings of the 9th international conference on Image Analysis and Recognition - Volume Part II
LNA: Fast Protein Structural Comparison Using a Laplacian Characterization of Tertiary Structure

IEEE/ACM Transactions on Computational Biology and Bioinformatics (TCBB)
Pattern classification of dermoscopy images: A perceptually uniform model

Pattern Recognition
Advanced probabilistic approach for network intrusion forecasting and detection

Expert Systems with Applications: An International Journal
Siblingrivalry: online autotuning through local competitions

Proceedings of the 2012 international conference on Compilers, architectures and synthesis for embedded systems
Texture based decision tree classification for Arecanut

Proceedings of the CUBE International Information Technology Conference
Prediction of flavin mono-nucleotide binding sites using modified PSSM profile and ensemble support vector machine

Computers in Biology and Medicine
Method to evaluate pose variability in automatic face recognition performance

International Journal of Biometrics
Annotating web images using NOVA: NOn-conVex group spArsity

Proceedings of the 20th ACM international conference on Multimedia
On the role of poetic versus nonpoetic features in “kindred” and diachronic poetry attribution

Journal of the American Society for Information Science and Technology
Evaluating question answering validation as a classification problem

Language Resources and Evaluation
Churn prediction in telecom using Random Forest and PSO based data balancing in combination with various feature selection strategies

Computers and Electrical Engineering
Improving ANNs performance on unbalanced data with an AUC-Based learning algorithm

ICANN'12 Proceedings of the 22nd international conference on Artificial Neural Networks and Machine Learning - Volume Part II
Design and Analysis of Classifier Learning Experiments in Bioinformatics: Survey and Case Studies

IEEE/ACM Transactions on Computational Biology and Bioinformatics (TCBB)
Improving Protein-Protein Interaction Pair Ranking with an Integrated Global Association Score

IEEE/ACM Transactions on Computational Biology and Bioinformatics (TCBB)
Sequence-Based Prediction of DNA-Binding Residues in Proteins with Conservation and Correlation Information

IEEE/ACM Transactions on Computational Biology and Bioinformatics (TCBB)
Learning spatial decision tree for geographical classification: a summary of results

Proceedings of the 20th International Conference on Advances in Geographic Information Systems
Permission-based abnormal application detection for android

ICICS'12 Proceedings of the 14th international conference on Information and Communications Security
Mode seeking clustering by KNN and mean shift evaluated

SSPR'12/SPR'12 Proceedings of the 2012 Joint IAPR international conference on Structural, Syntactic, and Statistical Pattern Recognition
Enhanced spatiotemporal relational probability trees and forests

Data Mining and Knowledge Discovery
Orthogonal support vector machine for credit scoring

Engineering Applications of Artificial Intelligence
ROC curve equivalence using the Kolmogorov-Smirnov test

Pattern Recognition Letters
Automatic recognition of quarantine citrus diseases

Expert Systems with Applications: An International Journal
Audience targeting by B-to-B advertisement classification: A neural network approach

Expert Systems with Applications: An International Journal
On the interplay of machine learning and background knowledge in image interpretation by Bayesian networks

Artificial Intelligence in Medicine
Serendipitous Personalized Ranking for Top-N Recommendation

WI-IAT '12 Proceedings of the The 2012 IEEE/WIC/ACM International Joint Conferences on Web Intelligence and Intelligent Agent Technology - Volume 01
A comparison of machine learning algorithms for proactive hard disk drive failure detection

Proceedings of the 4th international ACM Sigsoft symposium on Architecting critical systems
Metamorphic worm that carries its own morphing engine

Journal in Computer Virology
Feature space denoising improves word spotting

Proceedings of the 2nd International Workshop on Historical Document Imaging and Processing
An automatic computer-aided diagnosis system for liver tumours on computed tomography images

Computers and Electrical Engineering
Optimal level-crossing prediction for jump linear MIMO dynamical systems

Automatica (Journal of IFAC)
Customer attrition in retailing: An application of Multivariate Adaptive Regression Splines

Expert Systems with Applications: An International Journal
Set-oriented personalized ranking for diversified top-n recommendation

Proceedings of the 7th ACM conference on Recommender systems
Simple substitution distance and metamorphic detection

Journal in Computer Virology
ROC curves for regression

Pattern Recognition
EUSBoost: Enhancing ensembles for highly imbalanced data-sets by evolutionary undersampling

Pattern Recognition
Automatic classification for solitary pulmonary nodule in CT image by fractal analysis based on fractional Brownian motion model

Pattern Recognition
Chucky: exposing missing checks in source code for vulnerability discovery

Proceedings of the 2013 ACM SIGSAC conference on Computer & communications security
Early security classification of skype users via machine learning

Proceedings of the 2013 ACM workshop on Artificial intelligence and security
What's buzzing in the blizzard of buzz? Automotive component isolation in social media postings

Decision Support Systems
Quantitative security risk assessment of android permissions and applications

DBSec'13 Proceedings of the 27th international conference on Data and Applications Security and Privacy XXVII
Area under the distance threshold curve as an evaluation measure for probabilistic classifiers

MLDM'13 Proceedings of the 9th international conference on Machine Learning and Data Mining in Pattern Recognition
Bootstrap analysis of multiple repetitions of experiments using an interval-valued multiple comparison procedure

Journal of Computer and System Sciences
Automatic identification of experts and performance prediction in the multimodal math data corpus through analysis of speech interaction

Proceedings of the 15th ACM on International conference on multimodal interaction
Efficient distributed monitoring with active Collaborative Prediction

Future Generation Computer Systems
Variance inflation in high dimensional Support Vector Machines

Pattern Recognition Letters
An improved method of early diagnosis of smoking-induced respiratory changes using machine learning algorithms

Computer Methods and Programs in Biomedicine
A unified framework for reputation estimation in online rating systems

IJCAI'13 Proceedings of the Twenty-Third international joint conference on Artificial Intelligence
Integrated Fisher linear discriminants: An empirical study

Pattern Recognition
A Bayesian network model for predicting pregnancy after in vitro fertilization

Computers in Biology and Medicine
Automated cookie collection testing

ACM Transactions on Software Engineering and Methodology (TOSEM)
Automatic detection of Parkinsonism using significance measures and component analysis in DaTSCAN imaging

Neurocomputing
Addressing imbalanced classification with instance generation techniques: IPADE-ID

Neurocomputing
On the importance of the validation technique for classification with imbalanced datasets: Addressing covariate shift when data is skewed

Information Sciences: an International Journal
Multiobjective genetic programming for maximizing ROC performance

Neurocomputing
Exploiting the relationships among several binary classifiers via data transformation

Pattern Recognition
Half-AUC for the evaluation of sensitive or specific classifiers

Pattern Recognition Letters
To gather together for a better world: understanding and leveraging communities in micro-lending recommendation

Proceedings of the 23rd international conference on World wide web
CoBAn: A context based model for data leakage prevention

Information Sciences: an International Journal
Learning attribute weighted AODE for ROC area ranking

International Journal of Information and Communication Technology
A novel method for combining Bayesian networks, theoretical analysis, and its applications

Pattern Recognition
Online fault diagnosis method based on Incremental Support Vector Data Description and Extreme Learning Machine with incremental output structure

Neurocomputing
Estimation of a Priori Decision Threshold for Collocations Extraction: An Empirical Study

International Journal of Information Technology and Web Engineering
ROC analysis of classifiers in machine learning: A survey

Intelligent Data Analysis

Quantified Score

Hi-index	0.10

Visualization

Abstract

In this paper we investigate the use of the area under the receiver operating characteristic (ROC) curve (AUC) as a performance measure for machine learning algorithms. As a case study we evaluate six machine learning algorithms (C4.5, Multiscale Classifier, Perceptron, Multi-layer Perceptron, k-Nearest Neighbours, and a Quadratic Discriminant Function) on six ''real world'' medical diagnostics data sets. We compare and discuss the use of AUC to the more conventional overall accuracy and find that AUC exhibits a number of desirable properties when compared to overall accuracy: increased sensitivity in Analysis of Variance (ANOVA) tests; a standard error that decreased as both AUC and the number of test samples increased; decision threshold independent; and it is invariant to a priori class probabilities. The paper concludes with the recommendation that AUC be used in preference to overall accuracy for ''single number'' evaluation of machine learning algorithms.