Efficient similarity joins for near duplicate detection

  • Authors:

  • Chuan Xiao;Wei Wang;Xuemin Lin;Jeffrey Xu Yu

  • Affiliations:

  • University of New South Wales, Kensington, NSW, Australia;University of New South Wales, Kensington, NSW, Australia;University of New South Wales, Kensington, NSW, Australia;Chinese University of Hong Kong, Hong Kong, China

  • Venue:
  • Proceedings of the 17th international conference on World Wide Web
  • Year:
  • 2008

Quantified Score

Hi-index 0.00

Visualization

Abstract

With the increasing amount of data and the need to integrate data from multiple data sources, a challenging issue is to find near duplicate records efficiently. In this paper, we focus on efficient algorithms to find pairs of records such that their similarities are above a given threshold. Several existing algorithms rely on the prefix filtering principle to avoid computing similarity values for all possible pairs of records. We propose new filtering techniques by exploiting the ordering information; they are integrated into the existing methods and drastically reduce the candidate sizes and hence improve the efficiency. Experimental results show that our proposed algorithms can achieve up to 2.6x - 5x speed-up over previous algorithms on several real datasets and provide alternative solutions to the near duplicate Web page detection problem.