An empirical comparison of supervised learning algorithms

Authors:
Rich Caruana;Alexandru Niculescu-Mizil
Affiliations:
Cornell University, Ithaca, NY;Cornell University, Ithaca, NY
Venue:
ICML '06 Proceedings of the 23rd international conference on Machine learning
Year:
2006

Citing 12
Cited 84

The nature of statistical learning theory

The nature of statistical learning theory
Bagging predictors

Machine Learning
A Comparison of Prediction Accuracy, Complexity, and Training Time of Thirty-Three Old and New Classification Algorithms

Machine Learning
Random Forests

Machine Learning
An Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and Variants

Machine Learning
Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers

ICML '01 Proceedings of the Eighteenth International Conference on Machine Learning
Transforming classifier scores into accurate multiclass probability estimates

Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining
Tree Induction for Probability-Based Ranking

Machine Learning
Tree induction vs. logistic regression: a learning-curve analysis

The Journal of Machine Learning Research
Data mining in metric space: an empirical analysis of supervised learning performance criteria

Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining
Predicting good probabilities with supervised learning

ICML '05 Proceedings of the 22nd international conference on Machine learning
Data Mining: Practical Machine Learning Tools and Techniques, Second Edition (Morgan Kaufmann Series in Data Management Systems)

Data Mining: Practical Machine Learning Tools and Techniques, Second Edition (Morgan Kaufmann Series in Data Management Systems)

Classifying imbalanced data using a bagging ensemble variation (BEV)

ACM-SE 45 Proceedings of the 45th annual southeast regional conference
PAV and the ROC convex hull

Machine Learning
Toward knowledge-driven data mining

Proceedings of the 2007 international workshop on Domain driven data mining
Improving railroad wheel inspection planning using classification methods

AIAP'07 Proceedings of the 25th conference on Proceedings of the 25th IASTED International Multi-Conference: artificial intelligence and applications
Processing forecasting queries

VLDB '07 Proceedings of the 33rd international conference on Very large data bases
Boosting recombined weak classifiers

Pattern Recognition Letters
Non-stationary data sequence classification using online class priors estimation

Pattern Recognition
Agnostically learning decision trees

STOC '08 Proceedings of the fortieth annual ACM symposium on Theory of computing
Infobuttons and classification models: A method for the automatic selection of on-line information resources to fulfill clinicians' information needs

Journal of Biomedical Informatics
An empirical evaluation of supervised learning in high dimensions

Proceedings of the 25th international conference on Machine learning
Statistical diagnosis of unmodeled systematic timing effects

Proceedings of the 45th annual Design Automation Conference
A critical analysis of variants of the AUC

Machine Learning
When Overlapping Unexpectedly Alters the Class Imbalance Effects

IbPRIA '07 Proceedings of the 3rd Iberian conference on Pattern Recognition and Image Analysis, Part II
Generalizing Data in Natural Language

RSEISP '07 Proceedings of the international conference on Rough Sets and Intelligent Systems Paradigms
Classifier Loss Under Metric Uncertainty

ECML '07 Proceedings of the 18th European conference on Machine Learning
Feature Selection and Classification for Small Gene Sets

PRIB '08 Proceedings of the Third IAPR International Conference on Pattern Recognition in Bioinformatics
Learning from the Past with Experiment Databases

PRICAI '08 Proceedings of the 10th Pacific Rim International Conference on Artificial Intelligence: Trends in Artificial Intelligence
Classification models for the prediction of clinicians' information needs

Journal of Biomedical Informatics
Large scale multi-label classification via metalabeler

Proceedings of the 18th international conference on World wide web
Random Forest Classification for Automatic Delineation of Myocardium in Real-Time 3D Echocardiography

FIMH '09 Proceedings of the 5th International Conference on Functional Imaging and Modeling of the Heart
Out-of-bag estimation of the optimal sample size in bagging

Pattern Recognition
Automatic selection of high quality parses created by a fully unsupervised parser

CoNLL '09 Proceedings of the Thirteenth Conference on Computational Natural Language Learning
ODDboost: Incorporating Posterior Estimates into AdaBoost

MLDM '09 Proceedings of the 6th International Conference on Machine Learning and Data Mining in Pattern Recognition
A rose is a roos is a ruusu: querying translations for web image search

ACLShort '09 Proceedings of the ACL-IJCNLP 2009 Conference Short Papers
PLANET: massively parallel learning of tree ensembles with MapReduce

Proceedings of the VLDB Endowment
Evaluation of robustness and performance of early stopping rules with multi layer perceptrons

IJCNN'09 Proceedings of the 2009 international joint conference on Neural Networks
Margin-based Ranking and an Equivalence between AdaBoost and RankBoost

The Journal of Machine Learning Research
Image classification using marginalized kernels for graphs

GbRPR'07 Proceedings of the 6th IAPR-TC-15 international conference on Graph-based representations in pattern recognition
Analyzing PETs on imbalanced datasets when training and testing class distributions differ

PAKDD'08 Proceedings of the 12th Pacific-Asia conference on Advances in knowledge discovery and data mining
Combining clauses with various precisions and recalls to produce accurate probabilistic estimates

ILP'07 Proceedings of the 17th international conference on Inductive logic programming
Track-based self-supervised classification of dynamic obstacles

Autonomous Robots
Ensemble pruning via individual contribution ordering

Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining
Finding optimal classifiers for small feature sets in genomics and proteomics

Neurocomputing
Hunting for truly relevant articles in bioinformatics literature: a preliminary study

Proceedings of the First ACM International Conference on Bioinformatics and Computational Biology
Manifold learning for patient position detection in MRI

ISBI'10 Proceedings of the 2010 IEEE international conference on Biomedical imaging: from nano to Macro
Sensing foot gestures from the pocket

UIST '10 Proceedings of the 23nd annual ACM symposium on User interface software and technology
Improved fully unsupervised parsing with zoomed learning

EMNLP '10 Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing
Customer Validation of Commercial Predictive Models

Proceedings of the 2010 conference on Data Mining for Business Applications
A data mining framework for detecting subscription fraud in telecommunication

Engineering Applications of Artificial Intelligence
Data mining for credit card fraud: A comparative study

Decision Support Systems
Agreement-based semi-supervised learning for skull stripping

MICCAI'10 Proceedings of the 13th international conference on Medical image computing and computer-assisted intervention: Part III
Inactive learning?: difficulties employing active learning in practice

ACM SIGKDD Explorations Newsletter
Confidence driven unsupervised semantic parsing

HLT '11 Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1
Maximum likelihood for gaussians on graphs

GbRPR'11 Proceedings of the 8th international conference on Graph-based representations in pattern recognition
Identifying training sets for personalized article retrieval system

Proceedings of the 49th Annual Southeast Regional Conference
A Refined Margin Analysis for Boosting Algorithms via Equilibrium Margin

The Journal of Machine Learning Research
Vandalism detection in Wikipedia: a high-performing, feature-rich model and its reduction through Lasso

Proceedings of the 7th International Symposium on Wikis and Open Collaboration
Learning optical flow propagation strategies using random forests for fast segmentation in dynamic 2D & 3D echocardiography

MLMI'11 Proceedings of the Second international conference on Machine learning in medical imaging
Mass appraisal of residential apartments: An application of Random forest for valuation and a CART-based approach for model diagnostics

Expert Systems with Applications: An International Journal
On Equivalence Relationships Between Classification and Ranking Algorithms

The Journal of Machine Learning Research
Evolving neural networks with maximum AUC for imbalanced data classification

HAIS'10 Proceedings of the 5th international conference on Hybrid Artificial Intelligence Systems - Volume Part I
Editors Choice Article: I2VM: Incremental import vector machines

Image and Vision Computing
Learning probabilistic Description logic concepts: under different Assumptions on missing knowledge

Proceedings of the 27th Annual ACM Symposium on Applied Computing
Intelligent Postoperative Morbidity Prediction of Heart Disease Using Artificial Intelligence Techniques

Journal of Medical Systems
A survey of methods for data fusion and system adaptation using autonomic nervous system responses in physiological computing

Interacting with Computers
Content classification of development emails

Proceedings of the 34th International Conference on Software Engineering
Intelligible models for classification and regression

Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining
Random forests for metric learning with implicit pairwise position dependence

Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining
Towards automatically detecting whether student learning is shallow

ITS'12 Proceedings of the 11th international conference on Intelligent Tutoring Systems
An expert system for an innovative discrimination tool of commercial table grapes

ICIC'12 Proceedings of the 8th international conference on Intelligent Computing Theories and Applications
Mind the gap: learning to choose gaps for question generation

NAACL HLT '12 Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Learning speaker, addressee and overlap detection models from multimodal streams

Proceedings of the 14th ACM international conference on Multimodal interaction
Analyzing patient records to establish if and when a patient suffered from a medical condition

BioNLP '12 Proceedings of the 2012 Workshop on Biomedical Natural Language Processing
Using low-level dynamic attributes for malware detection based on data mining methods

MMM-ACNS'12 Proceedings of the 6th international conference on Mathematical Methods, Models and Architectures for Computer Network Security: computer network security
Embedding monte carlo search of features in tree-based ensemble methods

ECML PKDD'12 Proceedings of the 2012 European conference on Machine Learning and Knowledge Discovery in Databases - Volume Part I
Biomarker Identification and Cancer Classification Based on Microarray Data Using Laplace Naive Bayes Model with Mean Shrinkage

IEEE/ACM Transactions on Computational Biology and Bioinformatics (TCBB)
Batch-incremental versus instance-incremental learning in dynamic and evolving data

IDA'12 Proceedings of the 11th international conference on Advances in Intelligent Data Analysis
Annotating mobile phone location data with activity purposes using machine learning algorithms

Expert Systems with Applications: An International Journal
Partial Least Square Discriminant Analysis for bankruptcy prediction

Decision Support Systems
Randomness and sparsity induced codebook learning with application to cancer image classification

MCV'12 Proceedings of the Second international conference on Medical Computer Vision: recognition techniques and applications in medical imaging
Heterogeneous features and model selection for event-based media classification

Proceedings of the 3rd ACM conference on International conference on multimedia retrieval
Exerting Cost-Sensitive and Feature Creation Algorithms for Coronary Artery Disease Diagnosis

International Journal of Knowledge Discovery in Bioinformatics
A comparison of machine learning algorithms for proactive hard disk drive failure detection

Proceedings of the 4th international ACM Sigsoft symposium on Architecting critical systems
A survey on smartphone-based systems for opportunistic user context recognition

ACM Computing Surveys (CSUR)
A data mining approach for diagnosis of coronary artery disease

Computer Methods and Programs in Biomedicine
Comparative assessment of feature selection and classification techniques for visual inspection of pot plant seedlings

Computers and Electronics in Agriculture
On the doubt about margin explanation of boosting

Artificial Intelligence
Accurate probability calibration for multiple classifiers

IJCAI'13 Proceedings of the Twenty-Third international joint conference on Artificial Intelligence
Building simulation approaches for the training of automated data analysis tools in building energy management

Advanced Engineering Informatics
Editorial: Minimally-supervised learning of domain-specific causal relations using an open-domain corpus as knowledge base

Data & Knowledge Engineering
MetaStream: A meta-learning based method for periodic algorithm selection in time-changing data

Neurocomputing
Feature engineering for semantic place prediction

Pervasive and Mobile Computing
Reducing energy waste through eco-aware everyday things

Mobile Information Systems - Internet of Things
The impact of multinationality on firm value: A comparative analysis of machine learning techniques

Decision Support Systems

Quantified Score

Hi-index	0.00

Visualization

Abstract

A number of supervised learning methods have been introduced in the last decade. Unfortunately, the last comprehensive empirical evaluation of supervised learning was the Statlog Project in the early 90's. We present a large-scale empirical comparison between ten supervised learning methods: SVMs, neural nets, logistic regression, naive bayes, memory-based learning, random forests, decision trees, bagged trees, boosted trees, and boosted stumps. We also examine the effect that calibrating the models via Platt Scaling and Isotonic Regression has on their performance. An important aspect of our study is the use of a variety of performance criteria to evaluate the learning methods.