Efficient single-pass index construction for text databases

  • Authors:
  • Steffen Heinz;Justin Zobel

  • Affiliations:
  • School of Computer Science and Information Technology, RMIT University GPO Box 2476V, Melbourne 3001, Australia;School of Computer Science and Information Technology, RMIT University GPO Box 2476V, Melbourne 3001, Australia

  • Venue:
  • Journal of the American Society for Information Science and Technology
  • Year:
  • 2003

Quantified Score

Hi-index 0.00

Visualization

Abstract

Efficient construction of inverted indexes is essential to provision of search over large collections of text data. In this article, we review the principal approaches to inversion, analyze their theoretical cost, and present experimental results. We identify the drawbacks of existing inversion approaches and propose a single-pass inversion method that, in contrast to previous approaches, does not require the complete vocabulary of the indexed collection in main memory, can operate within limited resources, and does not sacrifice speed with high temporary storage requirements. We show that the performance of the single-pass approach can be improved by constructing inverted files in segments, reducing the cost of disk accesses during inversion of large volumes of data.