An effective algorithm for parallelizing sort merge joins in the presence of data skew

  • Authors:

  • Joel L. Wolf;Daniel M. Dias;Philip S. Yu

  • Affiliations:

  • P.O. Box 704, Yorktown Heights, N.Y. 10598, IBM Research Division, T. J. Watson Research Center;P.O. Box 704, Yorktown Heights, N.Y. 10598, IBM Research Division, T. J. Watson Research Center;P.O. Box 704, Yorktown Heights, N.Y. 10598, IBM Research Division, T. J. Watson Research Center

  • Venue:
  • DPDS '90 Proceedings of the second international symposium on Databases in parallel and distributed systems
  • Year:
  • 1990

Quantified Score

Hi-index 0.02

Visualization

Abstract

Parallel processing of relational queries has received considerable attention of late. However, in the presence of data skew, the speedup from conventional parallel join algorithms can be very limited, due to load imbalances among the various processors. Even a single large skew element can cause a processor to become overloaded. In this paper, we propose a parallel sort merge join algorithm which uses a divide-and-conquer approach to address the data skew problem. The proposed algorithm adds an extra scheduling phase to the usual sort, transfer and join phases. During the scheduling phase, a parallelizable optimization algorithm, using the output of the sort phase, attempts to balance the load across the multiple processors in the subsequent join phase. The algorithm naturally identifies the largest skew elements, and assigns each of them to an optimal number of processors. Assuming a Zipf-like distribution for data skew, the algorithm is demonstrated to achieve very good load balancing for the join phase in a CPU-bound environment, and is shown to be very robust relative to the degree of data skew and the total number of processors.