Web data integration using approximate string join

  • Authors:
  • Yingping Huang;Gregory Madey

  • Affiliations:
  • University of Notre Dame, Notre Dame, IN;University of Notre Dame, Notre Dame, IN

  • Venue:
  • Proceedings of the 13th international World Wide Web conference on Alternate track papers & posters
  • Year:
  • 2004

Quantified Score

Hi-index 0.00

Visualization

Abstract

Web data integration is an important preprocessing step for web mining. It is highly likely that several records on the web whose textual representations differ may represent the same real world entity. These records are called approximate duplicates. Data integration seeks to identify such approximate duplicates and merge them into integrated records. Many existing data integration algorithms make use of approximate string join, which seeks to (approximately) find all pairs of strings whose distances are less than a certain threshold. In this paper, we propose a new mapping method to detect pairs of strings with similarity above a certain threshold. In our method, each string is first mapped to a point in a high dimensional grid space, then pairs of points whose distances are 1 are identified. We implement it using Oracle SQL and PL/SQL. Finally, we evaluate this method using real data sets. Experimental results suggest that our method is both accurate and efficient.