Recent advances in error rate estimation
Pattern Recognition Letters
Elements of information theory
Elements of information theory
Beating the hold-out: bounds for K-fold and progressive cross-validation
COLT '99 Proceedings of the twelfth annual conference on Computational learning theory
Inference for the Generalization Error
Machine Learning
No Unbiased Estimator of the Variance of K-Fold Cross-Validation
The Journal of Machine Learning Research
Tutorial on Practical Prediction Theory for Classification
The Journal of Machine Learning Research
A comparison of tight generalization error bounds
ICML '05 Proceedings of the 22nd international conference on Machine learning
Artificial Intelligence in Medicine
Feature selection and classification model construction on type 2 diabetic patients' data
Artificial Intelligence in Medicine
An automated cervical pre-cancerous diagnostic system
Artificial Intelligence in Medicine
Artificial Intelligence in Medicine
Cross-validation and bootstrapping are unreliable in small sample classification
Pattern Recognition Letters
Guest editorial: Computational intelligence and machine learning in bioinformatics
Artificial Intelligence in Medicine
Cancer informatics by prototype networks in mass spectrometry
Artificial Intelligence in Medicine
Model of experts for decision support in the diagnosis of leukemia patients
Artificial Intelligence in Medicine
Artificial Intelligence in Medicine
Predicting breast cancer survivability: a comparison of three data mining methods
Artificial Intelligence in Medicine
Integrating classification trees with local logistic regression in Intensive Care prognosis
Artificial Intelligence in Medicine
Artificial Intelligence in Medicine
Artificial Intelligence in Medicine
Model selection for a medical diagnostic decision support system: a breast cancer detection case
Artificial Intelligence in Medicine
Bayesian mixed-effects inference on classification performance in hierarchical data sets
The Journal of Machine Learning Research
Hi-index | 0.00 |
Objective: Successful use of classifiers that learn to make decisions from a set of patient examples require robust methods for performance estimation. Recently many promising approaches for determination of an upper bound for the error rate of a single classifier have been reported but the Bayesian credibility interval (CI) obtained from a conventional holdout test still delivers one of the tightest bounds. The conventional Bayesian CI becomes unacceptably large in real world applications where the test set sizes are less than a few hundred. The source of this problem is that fact that the CI is determined exclusively by the result on the test examples. In other words, there is no information at all provided by the uniform prior density distribution employed which reflects complete lack of prior knowledge about the unknown error rate. Therefore, the aim of the study reported here was to study a maximum entropy (ME) based approach to improved prior knowledge and Bayesian CIs, demonstrating its relevance for biomedical research and clinical practice. Method and material: It is demonstrated how a refined non-uniform prior density distribution can be obtained by means of the ME principle using empirical results from a few designs and tests using non-overlapping sets of examples. Results: Experimental results show that ME based priors improve the CIs when employed to four quite different simulated and two real world data sets. Conclusions: An empirically derived ME prior seems promising for improving the Bayesian CI for the unknown error rate of a designed classifier.