Is Combining Classifiers with Stacking Better than Selecting the Best One?

Authors:
Saso Džeroski;Bernard Ženko
Affiliations:
Department of Knowledge Technologies, Jožef Stefan Institute, Jamova 39, SI-1000 Ljubljana, Slovenia. saso.dzeroski@ijs.si;Department of Knowledge Technologies, Jožef Stefan Institute, Jamova 39, SI-1000 Ljubljana, Slovenia.
Venue:
Machine Learning
Year:
2004

Citing 18
Cited 82

Instance-Based Learning Algorithms

Machine Learning
Original Contribution: Stacked generalization

Neural Networks
C4.5: programs for machine learning

C4.5: programs for machine learning
A schema for using multiple knowledge

Proceedings of the workshop on Computational learning theory and natural learning systems (vol. 2) : intersections between theory and experiment: intersections between theory and experiment
Approximate statistical tests for comparing supervised classification learning algorithms

Neural Computation
Data mining: practical machine learning tools and techniques with Java implementations

Data mining: practical machine learning tools and techniques with Java implementations
Using Model Trees for Classification

Machine Learning
Using Correspondence Analysis to Combine Classifiers

Machine Learning
A perspective view and survey of meta-learning

Artificial Intelligence Review
Combining Classifiers with Meta Decision Trees

Machine Learning
The Power of Decision Tables

ECML '95 Proceedings of the 8th European Conference on Machine Learning
Stacking with an Extended Set of Meta-level Attributes and MLR

ECML '02 Proceedings of the 13th European Conference on Machine Learning
Is Combining Classifiers Better than Selecting the Best One

ICML '02 Proceedings of the Nineteenth International Conference on Machine Learning
How to Make Stacking Better and Faster While Also Taking Care of an Unknown Weakness

ICML '02 Proceedings of the Nineteenth International Conference on Machine Learning
A Comparison of Stacking with Meta Decision Trees to Bagging, Boosting, and Stacking with other Methods

ICDM '01 Proceedings of the 2001 IEEE International Conference on Data Mining
Stacking with Multi-response Model Trees

MCS '02 Proceedings of the Third International Workshop on Multiple Classifier Systems
Issues in stacked generalization

Journal of Artificial Intelligence Research
Estimating continuous distributions in Bayesian classifiers

UAI'95 Proceedings of the Eleventh conference on Uncertainty in artificial intelligence

Introduction to the Special Issue on Meta-Learning

Machine Learning
Minority report in fraud detection: classification of skewed data

ACM SIGKDD Explorations Newsletter - Special issue on learning from imbalanced datasets
Learning to predict train wheel failures

Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining
On-line spam filter fusion

SIGIR '06 Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval
Combination of support vector machines using genetic programming

International Journal of Hybrid Intelligent Systems
Combining Information Extraction Systems Using Voting and Stacked Generalization

The Journal of Machine Learning Research
A cooperative constructive method for neural networks for pattern recognition

Pattern Recognition
Immune network based ensembles

Neurocomputing
Rough set Based Ensemble Classifier forWeb Page Classification

Fundamenta Informaticae
Non-strict heterogeneous Stacking

Pattern Recognition Letters
Pruning extensions to stacking

Intelligent Data Analysis
Selective fusion of heterogeneous classifiers

Intelligent Data Analysis
Comparing evolutionary hybrid systems for design and optimization of multilayer perceptron structure along training parameters

Information Sciences: an International Journal
Nonlinear Boosting Projections for Ensemble Construction

The Journal of Machine Learning Research
Distributed classification in peer-to-peer networks

Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining
Random Forests for multiclass classification: Random MultiNomial Logit

Expert Systems with Applications: An International Journal
The combination of multiple classifiers using an evidential reasoning approach

Artificial Intelligence
Neural Network Ensembles for Classification Problems Using Multiobjective Genetic Algorithms

ICANN '08 Proceedings of the 18th international conference on Artificial Neural Networks, Part I
Cascade RSVM in Peer-to-Peer Networks

ECML PKDD '08 Proceedings of the 2008 European Conference on Machine Learning and Knowledge Discovery in Databases - Part I
Combining Classifiers through Triplet-Based Belief Functions

ECML PKDD '08 Proceedings of the 2008 European Conference on Machine Learning and Knowledge Discovery in Databases - Part I
Boosting random subspace method

Neural Networks
Improving malware detection by applying multi-inducer ensemble

Computational Statistics & Data Analysis
A genetic encoding approach for learning methods for combining classifiers

Expert Systems with Applications: An International Journal
Pruning an ensemble of classifiers via reinforcement learning

Neurocomputing
Large scale multi-label classification via metalabeler

Proceedings of the 18th international conference on World wide web
Supervised projection approach for boosting classifiers

Pattern Recognition
Detection of malicious code by applying machine learning classifiers on static features: A state-of-the-art survey

Information Security Tech. Report
Intelligent file scoring system for malware detection from the gray list

Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining
Algorithm of Neural Network Ensembles and Robust Learning

ISNN '09 Proceedings of the 6th International Symposium on Neural Networks on Advances in Neural Networks
Unsupervised Hierarchical Weighted Multi-segmenter

MCS '09 Proceedings of the 8th International Workshop on Multiple Classifier Systems
On combining multiple classifiers using an evidential approach

AAAI'06 Proceedings of the 21st national conference on Artificial intelligence - Volume 1
Taxonomy for characterizing ensemble methods in classification tasks: A review and annotated bibliography

Computational Statistics & Data Analysis
A comparison of fraud cues and classification methods for fake escrow website detection

Information Technology and Management
Activity recognition from accelerometer data

IAAI'05 Proceedings of the 17th conference on Innovative applications of artificial intelligence - Volume 3
A fusion of stacking with dynamic integration

IJCAI'07 Proceedings of the 20th international joint conference on Artifical intelligence
Troika - An improved stacking schema for classification tasks

Information Sciences: an International Journal
Stacking recommendation engines with additional meta-features

Proceedings of the third ACM conference on Recommender systems
Meta-learning for imbalanced data and classification ensemble in binary classification

Neurocomputing
Constructing ensembles of classifiers by means of weighted instance selection

IEEE Transactions on Neural Networks
Two-stage classifications for improving time-to-failure estimates: a case study in prognostic of train wheels

Applied Intelligence
Ensemble-based classifiers

Artificial Intelligence Review
Group search optimizer: an optimization algorithm inspired by animal searching behavior

IEEE Transactions on Evolutionary Computation
GA-stacking: Evolutionary stacked generalization

Intelligent Data Analysis
Collaborative multi-agent rock facies classification from wireline well log data

Engineering Applications of Artificial Intelligence
Employing personal/impersonal views in supervised and semi-supervised sentiment classification

ACL '10 Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics
Sentiment classification and polarity shifting

COLING '10 Proceedings of the 23rd International Conference on Computational Linguistics
A meta learning approach: classification by cluster analysis

AIMSA'10 Proceedings of the 14th international conference on Artificial intelligence: methodology, systems, and applications
Stacking class probabilities obtained from view-based cluster ensembles

ICAISC'10 Proceedings of the 10th international conference on Artificial intelligence and soft computing: Part I
Statistical approaches to combining binary classifiers for multi-class classification

Neurocomputing
Ensemble of feature sets and classification algorithms for sentiment classification

Information Sciences: an International Journal
Reranking for stacking ensemble learning

ICONIP'10 Proceedings of the 17th international conference on Neural information processing: theory and algorithms - Volume Part I
Exploring the use of word relation features for sentiment classification

COLING '10 Proceedings of the 23rd International Conference on Computational Linguistics: Posters
Novel fusion methods for pattern recognition

ECML PKDD'11 Proceedings of the 2011 European conference on Machine learning and knowledge discovery in databases - Volume Part I
An experimental study of one- and two-level classifier fusion for different sample sizes

Pattern Recognition Letters
Classification by cluster analysis: a new meta-learning based approach

MCS'11 Proceedings of the 10th international conference on Multiple classifier systems
A case study of stacked multi-view learning in dementia research

AIME'11 Proceedings of the 13th conference on Artificial intelligence in medicine
Applying REC analysis to ensembles of sigma-point kalman filters

ICANN'06 Proceedings of the 16th international conference on Artificial Neural Networks - Volume Part II
Protein classification with multiple algorithms

PCI'05 Proceedings of the 10th Panhellenic conference on Advances in Informatics
Fingerprint authentication based on matching scores with other data

ICB'06 Proceedings of the 2006 international conference on Advances in Biometrics
StackTIS: A stacked generalization approach for effective prediction of translation initiation sites

Computers in Biology and Medicine
Error-sensitive grading for model combination

ECML'05 Proceedings of the 16th European conference on Machine Learning
On hybrid classification using model assisted posterior estimates

Pattern Recognition
Cascade ensembles

IWANN'05 Proceedings of the 8th international conference on Artificial Neural Networks: computational Intelligence and Bioinspired Systems
A group search optimizer for neural network training

ICCSA'06 Proceedings of the 2006 international conference on Computational Science and Its Applications - Volume Part III
Exploratory class-imbalanced and non-identical data distribution in automatic keyphrase extraction

ISNN'12 Proceedings of the 9th international conference on Advances in Neural Networks - Volume Part II
Predicting warfarin dosage from clinical data: A supervised learning approach

Artificial Intelligence in Medicine
Rough set Based Ensemble Classifier forWeb Page Classification

Fundamenta Informaticae
A meta learning approach to grammatical error correction

ACL '12 Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Short Papers - Volume 2
A weakly supervised model for sentence-level semantic orientation analysis with multiple experts

EMNLP-CoNLL '12 Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning
Improving reliability of oil spill detection systems using boosting for high-level feature selection

ICIAR'07 Proceedings of the 4th international conference on Image Analysis and Recognition
Minimal correlation classification

ECCV'12 Proceedings of the 12th European conference on Computer Vision - Volume Part VI
Artificial neural network training using a new efficient optimization algorithm

Applied Soft Computing
Assembling the optimal sentiment classifiers

WISE'12 Proceedings of the 13th international conference on Web Information Systems Engineering
Model selection based product kernel learning for regression on graphs

Proceedings of the 28th Annual ACM Symposium on Applied Computing
Outlier ensembles: position paper

ACM SIGKDD Explorations Newsletter
Metafraud: a meta-learning framework for detecting financial fraud

MIS Quarterly
Improving quality control by early prediction of manufacturing outcomes

Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining
Classification in P2P networks with cascade support vector machines

ACM Transactions on Knowledge Discovery from Data (TKDD)
Learning ensemble classifiers via restricted Boltzmann machines

Pattern Recognition Letters
MetaStream: A meta-learning based method for periodic algorithm selection in time-changing data

Neurocomputing
A framework for selection and fusion of pattern classifiers in multimedia recognition

Pattern Recognition Letters
Hybrid random subsample classifier ensemble for high dimensional data sets

International Journal of Hybrid Intelligent Systems

Quantified Score

Hi-index	0.00

Visualization

Abstract

We empirically evaluate several state-of-the-art methods for constructing ensembles of heterogeneous classifiers with stacking and show that they perform (at best) comparably to selecting the best classifier from the ensemble by cross validation. Among state-of-the-art stacking methods, stacking with probability distributions and multi-response linear regression performs best. We propose two extensions of this method, one using an extended set of meta-level features and the other using multi-response model trees to learn at the meta-level. We show that the latter extension performs better than existing stacking approaches and better than selecting the best classifier by cross validation.