On bit-parallel processing of multi-byte text

  • Authors:

  • Heikki Hyyrö;Jun Takaba;Ayumi Shinohara;Masayuki Takeda

  • Affiliations:

  • PRESTO, Japan Science and Technology Agency (JST);Department of Informatics, Kyushu University 33, Fukuoka, Japan;PRESTO, Japan Science and Technology Agency (JST);Department of Informatics, Kyushu University 33, Fukuoka, Japan

  • Venue:
  • AIRS'04 Proceedings of the 2004 international conference on Asian Information Retrieval Technology
  • Year:
  • 2004

Quantified Score

Hi-index 0.00

Visualization

Abstract

There exist practical bit-parallel algorithms for several types of pair-wise string processing, such as longest common subsequence computation or approximate string matching. The bit-parallel algorithms typically use a size-σ table of match bit-vectors, where the bits in the vector for a character λ identify the positions where the character λ occurs in one of the processed strings, and σ is the alphabet size. The time or space cost of computing the match table is not prohibitive with reasonably small alphabets such as ASCII text. However, for example in the case of general Unicode text the possible numerical code range of the characters is roughly one million. This makes using a simple table impractical. In this paper we evaluate three different schemes for overcoming this problem. First we propose to replace the character code table by a character code automaton. Then we compare this method with two other schemes: using a hash table, and the binary-search based solution proposed by Wu, Manber and Myers [25]. We find that the best choice is to use either the automaton-based method or a hash table.