Compression of concatenated web pages using XBW

  • Authors:
  • Radovan Šesták;Jan Lánský

  • Affiliations:
  • Charles University, Faculty of Mathematics and Physics, Department of Software Engineering, Praha 1, Czech Republic;Charles University, Faculty of Mathematics and Physics, Department of Software Engineering, Praha 1, Czech Republic

  • Venue:
  • SOFSEM'08 Proceedings of the 34th conference on Current trends in theory and practice of computer science
  • Year:
  • 2008

Quantified Score

Hi-index 0.00

Visualization

Abstract

XBW [10] is modular program for lossless compression that enables testing various combinations of algorithms. We obtained best results with XML parser creating dictionary of syllables or words combined with Burrows-Wheeler transform - hence the name XBW. The motivation for creating parser that handles non-valid XML and HTML files, has been system EGOTHOR [5] for full-text searching. On files of size approximately 20MB, formed by hundreds of web pages, we achieved twice the compression ratio of bzip2 while running only twice as long. For smaller files, XBWhas very good results, compared with other programs, especially for languages with rich morphology such as Slovak or German. For any big textual files, our program has good balance of compression and run time. Program XBW enables use of parser and coder with any implemented algorithm for compression.We have implemented Burrows-Wheeler transform which together with MTF and RLE forms block compression, dictionary methods LZC and LZSS, and finally statistical method PPM. Coder offers choice of Huffman and arithmetic coding.