Using discordance to improve classification in narrative clinical databases: An application to community-acquired pneumonia

Authors:
George Hripcsak;Charles Knirsch;Li Zhou;Adam Wilcox;Genevieve B. Melton
Affiliations:
Department of Biomedical Informatics, Columbia University, New York, NY 10032, USA;Pfizer Inc., New York, NY, USA;Department of Biomedical Informatics, Columbia University, New York, NY 10032, USA;Department of Biomedical Informatics, Columbia University, New York, NY 10032, USA;Department of Surgery, Johns Hopkins University, Baltimore, MD, USA
Venue:
Computers in Biology and Medicine
Year:
2007

Citing 10
Cited 0

Statistical analysis with missing data

Statistical analysis with missing data
Object-oriented development of a concept learning system for time-centered clinical data

Journal of Medical Systems
Data mining

Data mining
Advances in knowledge discovery and data mining

Advances in knowledge discovery and data mining
Coding neuroradiology reports for the northern Manhattan stroke study: a comparison of natural language processing and manual review

Computers and Biomedical Research
Challenging conventional assumptions of automated information retrieval with real users: Boolean searching and batch retrieval evaluations

Information Processing and Management: an International Journal - Special issue on interactivity at the text retrieval conference (TREC)
Imputation of Missing Data in Industrial Databases

Applied Intelligence
Towards the development of a conceptual distance metric for the UMLS

Journal of Biomedical Informatics
Extracting information on pneumonia in infants using natural language processing of radiology reports

Journal of Biomedical Informatics - Special section: JAMA commentaries
Self-organizing map for cluster analysis of a breast cancer database

Artificial Intelligence in Medicine

Quantified Score

Hi-index	0.00

Visualization

Abstract

Data mining in electronic medical records may facilitate clinical research, but much of the structured data may be miscoded, incomplete, or non-specific. The exploitation of narrative data using natural language processing may help, although nesting, varying granularity, and repetition remain challenges. In a study of community-acquired pneumonia using electronic records, these issues led to poor classification. Limiting queries to accurate, complete records led to vastly reduced, possibly biased samples. We exploited knowledge latent in the electronic records to improve classification. A similarity metric was used to cluster cases. We defined discordance as the degree to which cases within a cluster give different answers for some query that addresses a classification task of interest. Cases with higher discordance are more likely to be incorrectly classified, and can be reviewed manually to adjust the classification, improve the query, or estimate the likely accuracy of the query. In a study of pneumonia-in which the ICD9-CM coding was found to be very poor-the discordance measure was statistically significantly correlated with classification correctness (.45; 95% CI .15-.62).