SC spectra: a linear-time soft cardinality approximation for text comparison

  • Authors:
  • Sergio Jiménez Vargas;Alexander Gelbukh

  • Affiliations:
  • Intelligent Systems Research Laboratory (LISI), Systems and Industrial Engineering Department, National University of Colombia, Bogota, Colombia;Center for Computing Research (CIC), National Polytechnic Institute (IPN), Mexico City, Mexico

  • Venue:
  • MICAI'11 Proceedings of the 10th international conference on Artificial Intelligence: advances in Soft Computing - Volume Part II
  • Year:
  • 2011

Quantified Score

Hi-index 0.00

Visualization

Abstract

Soft cardinality (SC) is a softened version of the classical cardinality of set theory. However, given its prohibitive cost of computing (exponential order), an approximation that is quadratic in the number of terms in the text has been proposed in the past. SC Spectra is a new method of approximation in linear time for text strings, which divides text strings into consecutive substrings (i.e., q-grams) of different sizes. Thus, SC in combination with resemblance coefficients allowed the construction of a family of similarity functions for text comparison. These similarity measures have been used in the past to address a problem of entity resolution (name matching) outperforming SoftTFIDF measure. SC spectra method improves the previous results using less time and obtaining better performance. This allows the new method to be used with relatively large documents such as those included in classic information retrieval collections. SC spectra method exceeded SoftTFIDF and cosine tf-idf baselines with an approach that requires no term weighing.