Measuring the prediction error. A comparison of cross-validation, bootstrap and covariance penalty methods

Authors:
Simone Borra;Agostino Di Ciaccio
Affiliations:
Department of Economics and Territory, University of Rome "Tor Vergata", via Columbia 2, 00133, Italy;Department of Statistics, Probability and Appl. Statistics, University of Rome "La Sapienza", P.le Aldo Moro 5, 00185, Italy
Venue:
Computational Statistics & Data Analysis
Year:
2010

Citing 9
Cited 13

A bound on the error of cross validation using the approximation and estimation rates, with consequences for the training-test split

Neural Computation
Approximate statistical tests for comparing supervised classification learning algorithms

Neural Computation
Algorithmic stability and sanity-check bounds for leave-one-out cross-validation

Neural Computation
Resampling methods for variable selection in robust regression

Computational Statistics & Data Analysis
No Unbiased Estimator of the Variance of K-Fold Cross-Validation

The Journal of Machine Learning Research
Prediction error estimation: a comparison of resampling methods

Bioinformatics
Estimation of the conditional risk in classification: The swapping method

Computational Statistics & Data Analysis
Estimating classification error rate: Repeated cross-validation, repeated hold-out and bootstrap

Computational Statistics & Data Analysis
A study of cross-validation and bootstrap for accuracy estimation and model selection

IJCAI'95 Proceedings of the 14th international joint conference on Artificial intelligence - Volume 2

Editorial: Special issue on variable selection and robust procedures

Computational Statistics & Data Analysis
On the performance of the flexible maximum entropy distributions within partially adaptive estimation

Computational Statistics & Data Analysis
Investigation of bagging ensembles of genetic neural networks and fuzzy systems for real estate appraisal

ACIIDS'11 Proceedings of the Third international conference on Intelligent information and database systems - Volume Part II
Empirical comparison of resampling methods using genetic neural networks for a regression problem

HAIS'11 Proceedings of the 6th international conference on Hybrid artificial intelligent systems - Volume Part II
Experimental comparison of resampling methods in a multi-agent system to assist with property valuation

KES-AMSTA'11 Proceedings of the 5th KES international conference on Agent and multi-agent systems: technologies and applications
Empirical comparison of resampling methods using genetic fuzzy systems for a regression problem

IDEAL'11 Proceedings of the 12th international conference on Intelligent data engineering and automated learning
Wrapper feature selection for small sample size data driven by complete error estimates

Computer Methods and Programs in Biomedicine
eXiT*CBR.v2: Distributed case-based reasoning tool for medical prognosis

Decision Support Systems
Resampling methods for quality assessment of classifier performance and optimal number of features

Signal Processing
An innovative method for dynamic update of initial water table in XXT model based on neural network technique

Applied Soft Computing
Lazy overfitting control

MLDM'13 Proceedings of the 9th international conference on Machine Learning and Data Mining in Pattern Recognition
An Exponential Monte-Carlo algorithm for feature selection problems

Computers and Industrial Engineering
Detecting rare events using extreme value statistics applied to epileptic convulsions in children

Artificial Intelligence in Medicine

Quantified Score

Hi-index	0.03

Visualization

Abstract

The estimators most widely used to evaluate the prediction error of a non-linear regression model are examined. An extensive simulation approach allowed the comparison of the performance of these estimators for different non-parametric methods, and with varying signal-to-noise ratio and sample size. Estimators based on resampling methods such as Leave-one-out, parametric and non-parametric Bootstrap, as well as repeated Cross Validation methods and Hold-out, were considered. The methods used are Regression Trees, Projection Pursuit Regression and Neural Networks. The repeated-corrected 10-fold Cross-Validation estimator and the Parametric Bootstrap estimator obtained the best performance in the simulations.