Journal of Biomedical Informatics
Computational Statistics & Data Analysis
Computational Biology and Chemistry
Uncertainty estimation with a finite dataset in the assessment of classification models
Computational Statistics & Data Analysis
Classifier variability: Accounting for training and testing
Pattern Recognition
Expert Systems with Applications: An International Journal
A New Measure of Classifier Performance for Gene Expression Data
IEEE/ACM Transactions on Computational Biology and Bioinformatics (TCBB)
ACM Transactions on Knowledge Discovery from Data (TKDD) - Special Issue on the Best of SIGKDD 2011
Design and Analysis of Classifier Learning Experiments in Bioinformatics: Survey and Case Studies
IEEE/ACM Transactions on Computational Biology and Bioinformatics (TCBB)
Classifier Ensemble Methods for Diagnosing COPD from Volatile Organic Compounds in Exhaled Air
International Journal of Knowledge Discovery in Bioinformatics
Computers in Biology and Medicine
Hi-index | 3.84 |
Motivation: The receiver operator characteristic (ROC) curves are commonly used in biomedical applications to judge the performance of a discriminant across varying decision thresholds. The estimated ROC curve depends on the true positive rate (TPR) and false positive rate (FPR), with the key metric being the area under the curve (AUC). With small samples these rates need to be estimated from the training data, so a natural question arises: How well do the estimates of the AUC, TPR and FPR compare with the true metrics? Results: Through a simulation study using data models and analysis of real microarray data, we show that (i) for small samples the root mean square differences of the estimated and true metrics are considerable; (ii) even for large samples, there is only weak correlation between the true and estimated metrics; and (iii) generally, there is weak regression of the true metric on the estimated metric. For classification rules, we consider linear discriminant analysis, linear support vector machine (SVM) and radial basis function SVM. For error estimation, we consider resubstitution, three kinds of cross-validation and bootstrap. Using resampling, we show the unreliability of some published ROC results. Availability: Companion web site at http://compbio.tgen.org/paper_supp/ROC/roc.html Contact: edward@mail.ece.tamu.edu