Comparing inverted files and signature files for searching a large lexicon

  • Authors:

  • Ben Carterette;Fazli Can

  • Affiliations:

  • Center for Intelligent Information Retrieval, University of Massachusetts, Amherst, MA and Department of Computer Science and Systems Analysis, Miami University, Oxford;Department of Computer Science and Systems Analysis, Miami University, Oxford

  • Venue:
  • Information Processing and Management: an International Journal - Special issue: Cross-language information retrieval
  • Year:
  • 2005

Quantified Score

Hi-index 0.00

Visualization

Abstract

Signature files and inverted files are well-known index structures. In this paper we undertake a direct comparision of the two for searching for partially-specified queries in a large lexicon stored in main memory. Using n-grams to index lexicon terms, a bit-sliced signature file can be compressed to a smaller size than an inverted file if each n-gram sets only one bit in the term signature. With a signature width less than half the number of unique n-grams in the lexicon, the signature file method is about as fast as the inverted file method, and significantly smaller. Greater flexibility in memory usage and faster index generation time make signature files appropriate for searching large lexicons or other collections in an environment where memory is at a premium.