A Scalable Parallel Algorithm for Self-Organizing Maps with Applicationsto Sparse Data Mining Problems

  • Authors:
  • R. D. Lawrence;G. S. Almasi;H. E. Rushmeier

  • Affiliations:
  • IBM T.J. Watson Research Center, P.O. Box 218, Yorktown Heights, New York 10598. lawrence@watson.ibm.com;IBM T.J. Watson Research Center, P.O. Box 218, Yorktown Heights, New York 10598. almasi@watson.ibm.com;IBM T.J. Watson Research Center, P.O. Box 218, Yorktown Heights, New York 10598. holly@watson.ibm.com

  • Venue:
  • Data Mining and Knowledge Discovery
  • Year:
  • 1999

Quantified Score

Hi-index 0.00

Visualization

Abstract

We describe a scalable parallel implementation of the self organizing map (SOM) suitable for data-mining applications involving clustering or segmentation against large data sets such as those encountered in the analysis of customer spending patterns.The parallel algorithm is based on the batch SOM formulation in which the neural weights are updated at the end of each pass over the trainingdata. The underlying serial algorithm is enhanced to take advantage of the sparseness often encountered in these data sets. Analysis of a realistic test problem shows that the batch SOM algorithm captures key features observed using the conventional on-line algorithm,with comparable convergence rates.Performance measurements on an SP2 parallel computer are given for two retail data sets and a publicly available set of census data.These results demonstrate essentially linear speedup for the parallel batch SOM algorithm, using both a memory-contained sparse formulation as well as a separate implementation in which the mining data is accessed directly from a parallel file system. We also present visualizationsof the census data to illustrate the value of the clustering informationobtained via the parallel SOM method.