Fast Indexes and Algorithms for Set Similarity Selection Queries

  • Authors:
  • Marios Hadjieleftheriou;Amit Chandel;Nick Koudas;Divesh Srivastava

  • Affiliations:
  • AT&TLabs-Research, Florham Park, NJ 07932, USA. marioh@research.att.com;Department of Computer Science, University of Toronto, Toronto, ON M5S 2E4, Canada. amit@cs.toronto.edu;Department of Computer Science, University of Toronto, Toronto, ON M5S 2E4, Canada. koudas@cs.toronto.edu;AT&TLabs-Research, Florham Park, NJ 07932, USA. divesh@research.att.com

  • Venue:
  • ICDE '08 Proceedings of the 2008 IEEE 24th International Conference on Data Engineering
  • Year:
  • 2008

Quantified Score

Hi-index 0.00

Visualization

Abstract

Data collections often have inconsistencies that arise due to a variety of reasons, and it is desirable to be able to identify and resolve them efficiently. Set similarity queries are commonly used in data cleaning for matching similar data. In this work we concentrate on set similarity selection queries: Given a query set, retrieve all sets in a collection with similarity greater than some threshold. Various set similarity measures have been proposed in the past for data cleaning purposes. In this work we concentrate on weighted similarity functions like TF/IDF, and introduce variants that are well suited for set similarity selections in a relational database context. These variants have special semantic properties that can be exploited to design very efficient index structures and algorithms for answering queries efficiently. We present modifications of existing technologies to work for set similarity selection queries. We also introduce three novel algorithms based on the Threshold Algorithm, that exploit the semantic properties of the new similarity measures to achieve the best performance in theory and practice.