Efficiency of data structures for detecting overlaps in digital documents

  • Authors:
  • Krisztián Monostori;Arkady Zaslavsky;Heinz Schmidt

  • Affiliations:
  • Monash University, Melbourne, Australia;Monash University, Melbourne, Australia;Monash University, Melbourne, Australia

  • Venue:
  • ACSC '01 Proceedings of the 24th Australasian conference on Computer science
  • Year:
  • 2001

Quantified Score

Hi-index 0.00

Visualization

Abstract

This paper analyses the efficiency of different data structures for detecting overlap in digital documents. Most existing approaches use some hash function to reduce the space requirements for their indices of chunks. Since a hash function can produce the same value for different chunks, false matches are possible. In this paper we propose an algorithm that can be used for eliminating those false matches. This algorithm uses a suffix tree structure, which is space consuming. We define a modified suffix tree that only considers chunks starting at the beginning of words and we show how the algorithm can work on this structure. We can alternatively reduce space requirements of a suffix tree by converting it to a directed acyclic graph. We show that suffix link information can be preserved in this new structure and the matching statistics algorithm still works with those modifications that we propose.