Inverted file compression through document identifier reassignment

  • Authors:

  • Wann-Yun Shieh;Tien-Fu Chen;Jean Jyh-Jiun Shann;Chung-Ping Chung

  • Affiliations:

  • Department of Computer Science and Information Engineering, National Chiao Tung University, Hsinchu 300, Taiwan, ROC;Department of Computer Science and Information Engineering, National Chung Cheng University, Chiayi 621, Taiwan, ROC;Department of Computer Science and Information Engineering, National Chiao Tung University, Hsinchu 300, Taiwan, ROC;Department of Computer Science and Information Engineering, National Chiao Tung University, Hsinchu 300, Taiwan, ROC

  • Venue:
  • Information Processing and Management: an International Journal
  • Year:
  • 2003

Quantified Score

Hi-index 0.00

Visualization

Abstract

The inverted file is the most popular indexing mechanism for document search in an information retrieval system. Compressing an inverted file can greatly improve document search rate. Traditionally, the d-gap technique is used in the inverted file compression by replacing document identifiers with usually much smaller gap values. However, fluctuating gap values cannot be efficiently compressed by some well-known prefix-free codes. To smoothen and reduce the gap values, we propose a document-identifier reassignment algorithm. This reassignment is based on a similarity factor between documents. We generate a reassignment order for all documents according to the similarity to reassign closer identifiers to the documents having closer relationships. Simulation results show that the average gap values of sample inverted files can be reduced by 30%, and the compression rate of d-gapped inverted file with prefix-free codes can be improved by 15%.