Model selection and assessment for classification using validation

Authors:
Wojciech Jaworski
Affiliations:
Faculty of Mathematics, Informatics and Mechanics, Warsaw University, Warsaw, Poland
Venue:
RSFDGrC'05 Proceedings of the 10th international conference on Rough Sets, Fuzzy Sets, Data Mining, and Granular Computing - Volume Part I
Year:
2005

Citing 6
Cited 2

Effects of Sample Size in Classifier Design

IEEE Transactions on Pattern Analysis and Machine Intelligence
Estimation of Classifier Performance

IEEE Transactions on Pattern Analysis and Machine Intelligence
Machine learning, neural and statistical classification

Machine learning, neural and statistical classification
What Size Test Set Gives Good Error Rate Estimates?

IEEE Transactions on Pattern Analysis and Machine Intelligence
Pattern Classification (2nd Edition)

Pattern Classification (2nd Edition)
Bounds for validation

Fundamenta Informaticae

Rule Induction: Combining Rough Set and Statistical Approaches

RSCTC '08 Proceedings of the 6th International Conference on Rough Sets and Current Trends in Computing
Hybridization of rough sets and statistical learning theory

Transactions on rough sets XIII

Quantified Score

Hi-index	0.00

Visualization

Abstract

We address the problem of determination of the size of the test set which can can guarantee statistically significant results in classifier error estimation and in selection of the best classifier from a given set. We focus on the case of the 0-1 valued loss function and we provide one and two sides optimal bounds for Validation (known also as Hold-Out Estimate and Train-and-Test Method). We also calculate the smallest sample size, necessary for obtaining the bound for given estimation accuracy and reliability of estimation, and we present the results in tables. Finally, we propose strategies for classifier design using the bounds derived.