Epsilon grid order: an algorithm for the similarity join on massive high-dimensional data

  • Authors:
  • Christian Böhm;Bernhard Braunmüller;Florian Krebs;Hans-Peter Kriegel

  • Affiliations:
  • Institute for Computer Science, University of Munich, Oettingenstr. 67, D-80538 München, Germany;Institute for Computer Science, University of Munich, Oettingenstr. 67, D-80538 München, Germany;Institute for Computer Science, University of Munich, Oettingenstr. 67, D-80538 München, Germany;Institute for Computer Science, University of Munich, Oettingenstr. 67, D-80538 München, Germany

  • Venue:
  • SIGMOD '01 Proceedings of the 2001 ACM SIGMOD international conference on Management of data
  • Year:
  • 2001

Quantified Score

Hi-index 0.00

Visualization

Abstract

The similarity join is an important database primitive which has been successfully applied to speed up applications such as similarity search, data analysis and data mining. The similarity join combines two point sets of a multidimensional vector space such that the result contains all point pairs where the distance does not exceed a parameter &egr;. In this paper, we propose the Epsilon Grid Order, a new algorithm for determining the similarity join of very large data sets. Our solution is based on a particular sort order of the data points, which is obtained by laying an equi-distant grid with cell length &egr; over the data space and comparing the grid cells lexicographically. A typical problem of grid-based approaches such as MSJ or the &egr;-kdB-tree is that large portions of the data sets must be held simultaneously in main memory. Therefore, these approaches do not scale to large data sets. Our technique avoids this problem by an external sorting algorithm and a particular scheduling strategy during the join phase. In the experimental evaluation, a substantial improvement over competitive techniques is shown.