s-grams: Defining generalized n-grams for information retrieval

  • Authors:

  • Anni Järvelin;Antti Järvelin;Kalervo Järvelin

  • Affiliations:

  • University of Tampere, Department of Information Studies, FIN-33014 University of Tampere, Finland;University of Tampere, Department of Computer Sciences, FIN-33014 University of Tampere, Finland;University of Tampere, Department of Information Studies, FIN-33014 University of Tampere, Finland

  • Venue:
  • Information Processing and Management: an International Journal
  • Year:
  • 2007

Quantified Score

Hi-index 0.02

Visualization

Abstract

n-grams have been used widely and successfully for approximate string matching in many areas. s-grams have been introduced recently as an n-gram based matching technique, where di-grams are formed of both adjacent and non-adjacent characters. s-grams have proved successful in approximate string matching across language boundaries in Information Retrieval (IR). s-grams however lack precise definitions. Also their similarity comparison lacks precise definition. In this paper, we give precise definitions for both. Our definitions are developed in a bottom-up manner, only assuming character strings and elementary mathematical concepts. Extending established practices, we provide novel definitions of s-gram profiles and the L"1 distance metric for them. This is a stronger string proximity measure than the popular Jaccard similarity measure because Jaccard is insensitive to the counts of each n-gram in the strings to be compared. However, due to the popularity of Jaccard in IR experiments, we define the reduction of s-gram profiles to binary profiles in order to precisely define the (extended) Jaccard similarity function for s-grams. We also show that n-gram similarity/distance computations are special cases of our generalized definitions.