Missing data in medical databases: Impute, delete or classify?

Authors:
Federico Cismondi;André S. Fialho;Susana M. Vieira;Shane R. Reti;JoãO M. C. Sousa;Stan N. Finkelstein
Affiliations:
Massachusetts Institute of Technology, Engineering Systems Division, 77 Massachusetts Avenue, 02139 Cambridge, MA, USA and Technical University of Lisbon, Instituto Superior Técnico, Departme ...;Massachusetts Institute of Technology, Engineering Systems Division, 77 Massachusetts Avenue, 02139 Cambridge, MA, USA and Technical University of Lisbon, Instituto Superior Técnico, Departme ...;Technical University of Lisbon, Instituto Superior Técnico, Department of Mechanical Engineering, CIS/IDMEC - LAETA, Av. Rovisco Pais, 1049-001 Lisbon, Portugal;Division of Clinical Informatics, Department of Medicine, Beth Israel Deaconess Medical Centre, Harvard Medical School, Boston, MA, USA;Technical University of Lisbon, Instituto Superior Técnico, Department of Mechanical Engineering, CIS/IDMEC - LAETA, Av. Rovisco Pais, 1049-001 Lisbon, Portugal;Massachusetts Institute of Technology, Engineering Systems Division, 77 Massachusetts Avenue, 02139 Cambridge, MA, USA
Venue:
Artificial Intelligence in Medicine
Year:
2013

Citing 6
Cited 0

Data preparation for data mining

Data preparation for data mining
Feature Selection for Knowledge Discovery and Data Mining

Feature Selection for Knowledge Discovery and Data Mining
Decision tree search methods in fuzzy modeling and classification

International Journal of Approximate Reasoning
Visual Methods for Analyzing Time-Oriented Data

IEEE Transactions on Visualization and Computer Graphics
Missing data imputation using statistical and machine learning methods in a real breast cancer problem

Artificial Intelligence in Medicine
Knowledge-based approach to septic shock patient data using a neural network with trapezoidal activation functions

Artificial Intelligence in Medicine

Quantified Score

Hi-index	0.00

Visualization

Abstract

Background: The multiplicity of information sources for data acquisition in modern intensive care units (ICUs) makes the resulting databases particularly susceptible to missing data. Missing data can significantly affect the performance of predictive risk modeling, an important technique for developing medical guidelines. The two most commonly used strategies for managing missing data are to impute or delete values, and the former can cause bias, while the later can cause both bias and loss of statistical power. Objectives: In this paper we present a new approach for managing missing data in ICU databases in order to improve overall modeling performance. Methods: We use a statistical classifier followed by fuzzy modeling to more accurately determine which missing data should be imputed and which should not. We firstly develop a simulation test bed to evaluate performance, and then translate that knowledge using exactly the same database as previously published work by [13]. Results: In this work, test beds resulted in datasets with missing data ranging 10-50%. Using this new approach to missing data we are able to significantly improve modeling performance parameters such as accuracy of classifications by an 11%, sensitivity by 13%, and specificity by 10%, including also area under the receiver-operator curve (AUC) improvement of up to 13%. Conclusions: In this work, we improve modeling performance in a simulated test bed, and then confirm improved performance replicating previously published work by using the proposed approach for missing data classification. We offer this new method to other researchers who wish to improve predictive risk modeling performance in the ICU through advanced missing data management.