Statistical process control for validating a classification tree model for predicting mortality - A novel approach towards temporal validation

Authors:
Lilian Minne;Saeid Eslami;Nicolette de Keizer;Evert de Jonge;Sophia E. de Rooij;Ameen Abu-Hanna
Affiliations:
Academic Medical Center, Department of Medical Informatics, PO Box 22660, 1100 DD Amsterdam, The Netherlands;Academic Medical Center, Department of Medical Informatics, PO Box 22660, 1100 DD Amsterdam, The Netherlands;Academic Medical Center, Department of Medical Informatics, PO Box 22660, 1100 DD Amsterdam, The Netherlands;Leiden University Medical Center, Department of Intensive Care, PO Box 9600, 2300 RC Leiden, The Netherlands;Academic Medical Center, Department of Geriatrics, The Netherlands;Academic Medical Center, Department of Medical Informatics, PO Box 22660, 1100 DD Amsterdam, The Netherlands
Venue:
Journal of Biomedical Informatics
Year:
2012

Citing 2
Cited 0

The impact of changing populations on classifier performance

KDD '99 Proceedings of the fifth ACM SIGKDD international conference on Knowledge discovery and data mining
Machine Learning

Machine Learning

Quantified Score

Hi-index	0.00

Visualization

Abstract

Prediction models are postulated as useful tools to support tasks such as clinical decision making and benchmarking. In particular, classification tree models have enjoyed much interest in the Biomedical Informatics literature. However, their prospective predictive performance over the course of time has not been investigated. In this paper we suggest and apply statistical process control methods to monitor over more than 5years the prospective predictive performance of TM80+, one of the few classification-tree models published in the clinical literature. TM80+ is a model for predicting mortality among very elderly patients in the intensive care based on a multi-center dataset. We also inspect the predictive performance at the tree's leaves. This study provides important insights into patterns of (in)stability of the tree's performance and its ''shelf life''. The study underlies the importance of continuous validation of prognostic models over time using statistical tools and the timely recalibration of tree models.