Evaluating learning algorithms and classifiers

Authors:
Niklas Lavesson;Paul Davidsson
Affiliations:
Department Software and Systems Engineering, School of Engineering, Blekinge Institute of Technology, Box 520, SE-372 25 Ronneby, Sweden.;Department Software and Systems Engineering, School of Engineering, Blekinge Institute of Technology, Box 520, SE-372 25 Ronneby, Sweden
Venue:
International Journal of Intelligent Information and Database Systems
Year:
2007

Citing 24
Cited 2

Bootstrap Techniques for Error Estimation

IEEE Transactions on Pattern Analysis and Machine Intelligence
Overfitting Avoidance as Bias

Machine Learning
The nature of statistical learning theory

The nature of statistical learning theory
Evaluation and Selection of Biases in Machine Learning

Machine Learning - Special issue on bias evaluation and selection
Towards robust model selection using estimation and approximation error bounds

COLT '96 Proceedings of the ninth annual conference on Computational learning theory
Approximate statistical tests for comparing supervised classification learning algorithms

Neural Computation
Measure—based classifier performance evaluation

Pattern Recognition Letters - Special issue on pattern recognition in practice VI
A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems

Machine Learning
Machine Learning

Machine Learning
A Tutorial on Support Vector Machines for Pattern Recognition

Data Mining and Knowledge Discovery
The Role of Occam‘s Razor in Knowledge Discovery

Data Mining and Knowledge Discovery
A perspective view and survey of meta-learning

Artificial Intelligence Review
The Case against Accuracy Estimation for Comparing Induction Algorithms

ICML '98 Proceedings of the Fifteenth International Conference on Machine Learning
Artificial Intelligence: A Modern Approach

Artificial Intelligence: A Modern Approach
Pattern Classification (2nd Edition)

Pattern Classification (2nd Edition)
Introduction to the Special Issue on Meta-Learning

Machine Learning
Data mining in metric space: an empirical analysis of supervised learning performance criteria

Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining
Subspace Information Criterion for Model Selection

Neural Computation
Data Mining: Practical Machine Learning Tools and Techniques, Second Edition (Morgan Kaufmann Series in Data Management Systems)

Data Mining: Practical Machine Learning Tools and Techniques, Second Edition (Morgan Kaufmann Series in Data Management Systems)
Building Support Vector Machines with Reduced Classifier Complexity

The Journal of Machine Learning Research
Quantifying the impact of learning algorithm parameter tuning

AAAI'06 Proceedings of the 21st national conference on Artificial intelligence - Volume 1
Estimating the accuracy of learned concepts

IJCAI'93 Proceedings of the 13th international joint conference on Artifical intelligence - Volume 2
A study of cross-validation and bootstrap for accuracy estimation and model selection

IJCAI'95 Proceedings of the 14th international joint conference on Artificial intelligence - Volume 2
A new metric-based approach to model selection

AAAI'97/IAAI'97 Proceedings of the fourteenth national conference on artificial intelligence and ninth conference on Innovative applications of artificial intelligence

Distributed learning with data reduction

Transactions on computational collective intelligence IV
A method for evaluation of learning components

Automated Software Engineering

Quantified Score

Hi-index	0.00

Visualization

Abstract

We analyse 18 evaluation methods for learning algorithms and classifiers, and show how to categorise these methods with the help of an evaluation method taxonomy based on several criteria. We also define a formal framework that make it possible to describe all methods using the same terminology, and apply it in a review of the state-of-the-art in learning algorithm and classifier evaluation. The framework enables comparison and deeper understanding of evaluation methods from different fields of research. Moreover, we argue that the framework and taxonomy support the process of finding candidate evaluation methods for a particular problem.