Induction with randomization testing: decision-oriented analysis of large data sets
Induction with randomization testing: decision-oriented analysis of large data sets
Using a Permutation Test for Attribute Selection in Decision Trees
ICML '98 Proceedings of the Fifteenth International Conference on Machine Learning
Data Mining: Practical Machine Learning Tools and Techniques, Second Edition (Morgan Kaufmann Series in Data Management Systems)
Selection of relevant genes in cancer diagnosis based on their prediction accuracy
Artificial Intelligence in Medicine
Assessing data mining results via swap randomization
ACM Transactions on Knowledge Discovery from Data (TKDD)
Cross-validation and bootstrapping are unreliable in small sample classification
Pattern Recognition Letters
Randomization methods for assessing data analysis results on real-valued matrices
Statistical Analysis and Data Mining
Permutation Tests for Studying Classifier Performance
ICDM '09 Proceedings of the 2009 Ninth IEEE International Conference on Data Mining
Support vector machine for functional data classification
Neurocomputing
Permutation tests for classification
COLT'05 Proceedings of the 18th annual conference on Learning Theory
Controlled permutations for testing adaptive classifiers
DS'11 Proceedings of the 14th international conference on Discovery science
ICIAR'12 Proceedings of the 9th international conference on Image Analysis and Recognition - Volume Part I
Recognizing facial expressions using a novel shape motion descriptor
Proceedings of the Eighth Indian Conference on Computer Vision, Graphics and Image Processing
Hi-index | 0.00 |
We explore the framework of permutation-based p-values for assessing the performance of classifiers. In this paper we study two simple permutation tests. The first test assess whether the classifier has found a real class structure in the data; the corresponding null distribution is estimated by permuting the labels in the data. This test has been used extensively in classification problems in computational biology. The second test studies whether the classifier is exploiting the dependency between the features in classification; the corresponding null distribution is estimated by permuting the features within classes, inspired by restricted randomization techniques traditionally used in statistics. This new test can serve to identify descriptive features which can be valuable information in improving the classifier performance. We study the properties of these tests and present an extensive empirical evaluation on real and synthetic data. Our analysis shows that studying the classifier performance via permutation tests is effective. In particular, the restricted permutation test clearly reveals whether the classifier exploits the interdependency between the features in the data.