Incremental all pairs similarity search for varying similarity thresholds

  • Authors:

  • Amit Awekar;Nagiza F. Samatova;Paul Breimyer

  • Affiliations:

  • North Carolina State University, Raleigh, NC and Oak Ridge National Laboratory, Oak Ridge, TN;North Carolina State University, Raleigh, NC and Oak Ridge National Laboratory, Oak Ridge, TN;North Carolina State University, Raleigh, NC and Oak Ridge National Laboratory, Oak Ridge, TN

  • Venue:
  • Proceedings of the 3rd Workshop on Social Network Mining and Analysis
  • Year:
  • 2009

Quantified Score

Hi-index 0.01

Visualization

Abstract

All Pairs Similarity Search (APSS) is a ubiquitous problem in many data mining applications and involves finding all pairs of records with similarity scores above a specified threshold. In this paper, we introduce the problem of Incremental All Pairs Similarity Search (IAPSS), where APSS is performed multiple times over the same dataset by varying the similarity threshold. To the best of our knowledge, this is the first work that addresses the IAPSS problem. All existing solutions for APSS perform redundant computations by invoking APSS independently for each threshold value. In contrast, our solution to the IAPSS problem avoids redundant computations by storing the history of previous APSS invocations and using index splitting. While offering obvious benefits, the computation and I/O intensive nature of the IAPSS solution raises two key research challenges: (1) to develop efficient I/O techniques to manage computation history and (2) to efficiently identify and prune redundant computations. We address these challenges through the proposed (a) history binning technique that clusters record pairs based on similarity values and performs I/O during the similarity computation, and (b) splitting of inverted index that maps each dimension to a list of records that have a non-zero projection along that dimension. As a result, we evaluate the effectiveness of our techniques by demonstrating speed-ups in the order of 2X to over 105 X over the state-of-the-art APSS algorithm for four real-world large-scale datasets.