Bootstrap analysis of multiple repetitions of experiments using an interval-valued multiple comparison procedure

Authors:
José Otero;Luciano Sánchez;Inés Couso;Ana Palacios
Affiliations:
Universidad de Oviedo, Computer Science Department, Spain;Universidad de Oviedo, Computer Science Department, Spain;Universidad de Oviedo, Statistics Department, Spain;Universidad de Granada, Computer Science Department, Spain
Venue:
Journal of Computer and System Sciences
Year:
2014

Citing 12
Cited 0

C4.5: programs for machine learning

C4.5: programs for machine learning
On Comparing Classifiers: Pitfalls toAvoid and a Recommended Approach

Data Mining and Knowledge Discovery
Bootstrap Methods for the Cost-Sensitive Evaluation of Classifiers

ICML '00 Proceedings of the Seventeenth International Conference on Machine Learning
Imprecise distribution function associated to a random set

Information Sciences—Informatics and Computer Science: An International Journal
Permutation, Parametric, and Bootstrap Tests of Hypotheses (Springer Series in Statistics)

Permutation, Parametric, and Bootstrap Tests of Hypotheses (Springer Series in Statistics)
Statistical Comparisons of Classifiers over Multiple Data Sets

The Journal of Machine Learning Research
KEEL: a software tool to assess evolutionary algorithms for data mining problems

Soft Computing - A Fusion of Foundations, Methodologies and Applications - Special Issue on Evolutionary and Metaheuristics based Data Mining (EMBDM); Guest Editors: José A. Gámez, María J. del Jesús, José M. Puerta
Pattern Recognition and Neural Networks

Pattern Recognition and Neural Networks
A study of cross-validation and bootstrap for accuracy estimation and model selection

IJCAI'95 Proceedings of the 14th international joint conference on Artificial intelligence - Volume 2
The use of the area under the ROC curve in the evaluation of machine learning algorithms

Pattern Recognition
Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: Experimental analysis of power

Information Sciences: an International Journal
Mark-recapture techniques in statistical tests for imprecise data

International Journal of Approximate Reasoning

Quantified Score

Hi-index	0.00

Visualization

Abstract

A new bootstrap test is introduced that allows for assessing the significance of the differences between stochastic algorithms in a cross-validation with repeated folds experimental setup. Intervals are used for modeling the variability of the data that can be attributed to the repetition of learning and testing stages over the same folds in cross validation. Numerical experiments are provided that support the following three claims: (1) Bootstrap tests can be more powerful than ANOVA or Friedman test for comparing multiple classifiers. (2) In the presence of outliers, interval-valued bootstrap tests achieve a better discrimination between stochastic algorithms than nonparametric tests. (3) Choosing ANOVA, Friedman or Bootstrap can produce different conclusions in experiments involving actual data from machine learning tasks.