A grid-based approach for enterprise-scale data mining

  • Authors:
  • Ramesh Natarajan;Radu Sion;Thomas Phan

  • Affiliations:
  • IBM Thomas J. Watson Research Center, Yorktown Heights, NY;Department of Computer Science, State University of New York, Stonybrook, NY;IBM Almaden Research Center, San Jose, CA

  • Venue:
  • Future Generation Computer Systems - Special section: Data mining in grid computing environments
  • Year:
  • 2007

Quantified Score

Hi-index 0.00

Visualization

Abstract

We describe a grid-based approach for enterprise-scale data mining, which is based on leveraging parallel database technology for data storage, and on-demand compute servers for parallelism in the statistical computations. This approach is targeted towards the use of data mining in highly-automated vertical business applications, where the data is stored on one or more relational database systems, and an independent set of high-performance compute servers or a network of low-cost, commodity processors is used to improve the application performance and overall workload management. The goal of this paper is to describe an algorithmic decomposition of data mining kernels between the data storage and compute grids, which makes it possible to exploit the parallelism on the respective grids in a simple way, while minimizing the data transfer between these grids. This approach is compatible with existing standards for data mining task specification and results reporting, so that larger applications using these data mining algorithms do not have to be modified to benefit from this grid-based approach.