Efficient similarity joins for near-duplicate detection

  • Authors:

  • Chuan Xiao;Wei Wang;Xuemin Lin;Jeffrey Xu Yu;Guoren Wang

  • Affiliations:

  • The University of New South Wales, Australia;The University of New South Wales, Australia;The University of New South Wales, Australia and East China Normal University;Chinese University of Hong Kong, Hong Kong;Northeastern University, China

  • Venue:
  • ACM Transactions on Database Systems (TODS)
  • Year:
  • 2011

Quantified Score

Hi-index 0.00

Visualization

Abstract

With the increasing amount of data and the need to integrate data from multiple data sources, one of the challenging issues is to identify near-duplicate records efficiently. In this article, we focus on efficient algorithms to find a pair of records such that their similarities are no less than a given threshold. Several existing algorithms rely on the prefix filtering principle to avoid computing similarity values for all possible pairs of records. We propose new filtering techniques by exploiting the token ordering information; they are integrated into the existing methods and drastically reduce the candidate sizes and hence improve the efficiency. We have also studied the implementation of our proposed algorithm in stand-alone and RDBMS-based settings. Experimental results show our proposed algorithms can outperform previous algorithms on several real datasets.