Technical Note: Selecting a Classification Method by Cross-Validation

Authors:
Cullen Schaffer
Affiliations:
Department of Computer Science, CUNY/Hunter College, 695 Park Avenue, New York, NY 10021. SCHAFFER@MARNA.HUNTER.CUNY.EDU
Venue:
Machine Learning
Year:
1993

Citing 0
Cited 37

Feature generation for sequence categorization

AAAI '98/IAAI '98 Proceedings of the fifteenth national/tenth conference on Artificial intelligence/Innovative applications of artificial intelligence
Machine Learning for Information Extraction in Informal Domains

Machine Learning - Special issue on information retrieval
Cascade Generalization

Machine Learning
Ranking Learning Algorithms: Using IBL and Meta-Learning on Accuracy and Time Results

Machine Learning
Improved Dataset Characterisation for Meta-learning

DS '02 Proceedings of the 5th International Conference on Discovery Science
An Evaluation of Grading Classifiers

IDA '01 Proceedings of the 4th International Conference on Advances in Intelligent Data Analysis
Using genetic programming to classify node positive patients in bladder cancer

Proceedings of the 8th annual conference on Genetic and evolutionary computation
Effective classification of noisy data streams with attribute-oriented dynamic classifier selection

Knowledge and Information Systems
Local averaging of heterogeneous regression models

International Journal of Hybrid Intelligent Systems
Local Feature Selection with Dynamic Integration of Classifiers

Fundamenta Informaticae - Intelligent Systems
An integrated approach for operational knowledge acquisition of refuse incinerators

Expert Systems with Applications: An International Journal
Local voting of weak classifiers

International Journal of Knowledge-based and Intelligent Engineering Systems
Dynamic integration of classifiers for handling concept drift

Information Fusion
How to shift bias: Lessons from the baldwin effect

Evolutionary Computation
Credit risk analysis using a hybrid data mining model

International Journal of Intelligent Systems Technologies and Applications
Feature Extraction for Dynamic Integration of Classifiers

Fundamenta Informaticae
Locally application of cascade generalization for classification problems

Intelligent Decision Technologies
A Case-Based Methodology for Feature Weighting Algorithm Recommendation

Proceedings of the 2005 conference on Artificial Intelligence Research and Development
Taxonomy for characterizing ensemble methods in classification tasks: A review and annotated bibliography

Computational Statistics & Data Analysis
Improved heterogeneous distance functions

Journal of Artificial Intelligence Research
Connectionist theory refinement: genetically searching the space of network topologies

Journal of Artificial Intelligence Research
Cost-sensitive classification: empirical evaluation of a hybrid genetic decision tree induction algorithm

Journal of Artificial Intelligence Research
Sequential genetic search for ensemble feature selection

IJCAI'05 Proceedings of the 19th international joint conference on Artificial intelligence
Ensemble-based classifiers

Artificial Intelligence Review
RWS (random walk splitting): a random walk based discretization of continuous attributes

PRICAI'00 Proceedings of the 6th Pacific Rim international conference on Artificial intelligence
The data complexity index to construct an efficient cross-validation method

Decision Support Systems
Lazy decision trees

AAAI'96 Proceedings of the thirteenth national conference on Artificial intelligence - Volume 1
Design and evaluation of neural networks for an embedded application

IEA/AIE'10 Proceedings of the 23rd international conference on Industrial engineering and other applications of applied intelligent systems - Volume Part III
Dynamic integration with random forests

ECML'06 Proceedings of the 17th European conference on Machine Learning
iASA: learning to annotate the semantic web

Journal on Data Semantics IV
A landmarker selection algorithm based on correlation and efficiency criteria

AI'04 Proceedings of the 17th Australian joint conference on Advances in Artificial Intelligence
Detecting effective connectivity in networks of coupled neuronal oscillators

Journal of Computational Neuroscience
Feature Extraction for Dynamic Integration of Classifiers

Fundamenta Informaticae
Local Feature Selection with Dynamic Integration of Classifiers

Fundamenta Informaticae - Intelligent Systems
Classification of Unseen Examples under Uncertainty

Fundamenta Informaticae
Automated identification of normal and diabetes heart rate signals using nonlinear measures

Computers in Biology and Medicine
Learning to filter spam emails: An ensemble learning approach

International Journal of Hybrid Intelligent Systems

Quantified Score

Hi-index	0.00

Visualization

Abstract

If we lack relevant problem-specific knowledge, cross-validation methods may be used to select a classification method empirically. We examine this idea here to show in what senses cross-validation does and does not solve the selection problem. As illustrated empirically, cross-validation may lead to higher average performance than application of any single classification strategy, and it also cuts the risk of poor performance. On the other hand, cross-validation is no more or less a form of bias than simpler strategies, and applying it appropriately ultimately depends in the same way on prior knowledge. In fact, cross-validation may be seen as a way of applying partial information about the applicability of alternative classification strategies.