An introduction to ROC analysis
Pattern Recognition Letters - Special issue: ROC analysis in pattern recognition
The Need for Open Source Software in Machine Learning
The Journal of Machine Learning Research
| Hi-index | 0.00 |
Determining diagnostic criteria for specific disorders is often a tedious task that involves determining optimal diagnostic thresholds for symptoms and biomarkers using receiver-operating characteristic (ROC) statistics. To help this endeavor, we developed softROC, a user-friendly graphic-based tool that lets users visually explore possible ROC tradeoffs. The software requires MATLAB installation and an Excel file containing threshold symptoms/biological measures, with corresponding gold standard diagnoses for a set of patients. The software scans the input file for diagnostic and symptom/biomarkers columns, and populates the graphical-user-interface (GUI). Users select symptoms/biomarkers of interest using Boolean algebra as potential inputs to create diagnostic criteria outputs. The software evaluates subtests across the user-established range of cut-points and compares them to a gold standard in order to generate ROC and quality ROC scatter plots. These plots can be examined interactively to find optimal cut-points of interest for a given application (e.g. sensitivity versus specificity needs). Split-set validation can also be used to set up criteria and validate these in independent samples. Bootstrapping is used to produce confidence intervals. Additional statistics and measures are provided, such as the area under the ROC curve (AUC). As a testing set, softROC is used to investigate nocturnal polysomnogram measures as diagnostic features for narcolepsy. All measures can be outputted to a text file for offline analysis. The softROC toolbox, with clinical training data and tutorial instruction manual, is provided as supplementary material and can be obtained online at http://www.stanford.edu/~hyatt4/software/softroc or from the open source repository at http://www.github.com/informaton/softroc.