Substring Count Estimation in Extremely Long Strings*This work was supported in part by the Brain Korea 21 Project and in part by the Ministry of Information & Communications, Korea, under the Information Technology Research Center (ITRC) Support Program in 2005.

  • Authors:

  • Jinuk Bae;Sukho Lee

  • Affiliations:

  • The authors are with the School of Computer Science and Engineering, Seoul National University, Seoul, Korea. E-mail: jinuk@db.snu.ac.kr;The authors are with the School of Computer Science and Engineering, Seoul National University, Seoul, Korea. E-mail: jinuk@db.snu.ac.kr

  • Venue:
  • IEICE - Transactions on Information and Systems
  • Year:
  • 2006

Quantified Score

Hi-index 0.00

Visualization

Abstract

To estimate the number of substring matches against string data, count suffix trees (CS-tree) have been used as a kind of alphanumeric histograms. Although the trees are useful for substring count estimation in short data strings (e.g. name or title), they reveal several drawbacks when the target is changed to extremely long strings. First, it becomes too hard or at least slow to build CS-trees, because their origin, the suffix tree, has memory-bottleneck problem with long strings. Secondly, some of CS-tree-node counts are incorrect due to frequent pruning of nodes. Therefore, we propose the count q-gram tree (CQ-tree) as an alphanumeric histogram for long strings. By adopting q-grams (or length-q substrings), CQ-trees can be created fast and correctly within small available memory. Furthermore, we mathematically provide the lower and upper bounds that the count estimation can reach to. To the best of our knowledge, our work is the first one to present such bounds among research activities to estimate the alphanumeric selectivity. Our experimental study shows that the CQ-tree outperforms the CS-tree in terms of the building time and accuracy.