Fully Dynamic Partitioning: Handling Data Skew in Parallel Data Cube Computation

  • Authors:

  • Hongjun Lu;Jeffrey Xu Yu;Ling Feng;Zhixian Li

  • Affiliations:

  • School of Computing, The National University of Singapore, Singapore, Republic of Singapore. luhj@comp.nus.edu.sg;Department of Systems Engineering and Engineering Management, The Chinese University of Hong Kong, Hong Kong, People's Republic of China. yu@se.cuhk.edu.hk;InfoLab, Tilburg University, The Netherlands. ling@kub.nl;School of Computing, The National University of Singapore, Singapore, Republic of Singapore. lizhixia@comp.nus.edu.sg

  • Venue:
  • Distributed and Parallel Databases
  • Year:
  • 2003

Quantified Score

Hi-index 0.00

Visualization

Abstract

Parallel data processing is a promising approach for efficiently computing data cube in relational databases, because most aggregate functions used in OLAP (On-Line Analytical Processing) are distributive functions. This paper studies the issues of handling data skew in parallel data cube computation. We present a fully dynamic partitioning approach that can effectively distribute workload among processing nodes without priori knowledge of data distribution. As supplement, a simple and effective dynamic load balancing mechanism is also incorporated into our algorithm, which further improves the overall performance. Our experimental results indicated that the proposed techniques are effective even when high data skew exists. The results of scale-up and speedup tests are also satisfactory.