Medical data mining: insights from winning two competitions

  • Authors:
  • Saharon Rosset;Claudia Perlich;Grzergorz Świrszcz;Prem Melville;Yan Liu

  • Affiliations:
  • School of Mathematical Sciences, Tel Aviv University, Tel Aviv, Israel 69978;IBM T.J. Watson Research Center, Yorktown Heights, USA 10598;IBM T.J. Watson Research Center, Yorktown Heights, USA 10598;IBM T.J. Watson Research Center, Yorktown Heights, USA 10598;IBM T.J. Watson Research Center, Yorktown Heights, USA 10598

  • Venue:
  • Data Mining and Knowledge Discovery
  • Year:
  • 2010

Quantified Score

Hi-index 0.00

Visualization

Abstract

Two major data mining competitions in 2008 presented challenges in medical domains: KDD Cup 2008, which concerned cancer detection from mammography data; and Informs Data Mining Challenge 2008, dealing with diagnosis of pneumonia based on patient information from hospital files. Our team won both of these competitions, and in this paper we share our lessons learned and insights. We emphasize the aspects that pertain to the general practice and methodology of medical data mining, rather than to the specifics of each modeling competition. We concentrate on three topics: information leakage, its effect on competitions and proof-of-concept projects; consideration of real-life model performance measures in model construction and evaluation; and relational learning approaches to medical data mining tasks.