A fast algorithm for optimal length-limited Huffman codes
Journal of the ACM (JACM)
Query evaluation techniques for large databases
ACM Computing Surveys (CSUR)
ACM SIGMOD Record
Data compression in full-text retrieval systems
Journal of the American Society for Information Science
Adding compression to a full-text retrieval system
Software—Practice & Experience
The art of computer programming, volume 3: (2nd ed.) sorting and searching
The art of computer programming, volume 3: (2nd ed.) sorting and searching
Managing gigabytes (2nd ed.): compressing and indexing documents and images
Managing gigabytes (2nd ed.): compressing and indexing documents and images
The implementation and performance of compressed databases
ACM SIGMOD Record
Query optimization in compressed database systems
SIGMOD '01 Proceedings of the 2001 ACM SIGMOD international conference on Management of data
Database System Implementation
Database System Implementation
Database Management Systems
Compression and Coding Algorithms
Compression and Coding Algorithms
Compression of inverted indexes For fast query evaluation
SIGIR '02 Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval
A general-purpose compression scheme for large collections
ACM Transactions on Information Systems (TOIS)
Text Compression for Dynamic Document Databases
IEEE Transactions on Knowledge and Data Engineering
Relational Database Compression Using Augmented Vector Quantization
ICDE '95 Proceedings of the Eleventh International Conference on Data Engineering
Compressing Relations and Indexes
ICDE '98 Proceedings of the Fourteenth International Conference on Data Engineering
Database Architecture Optimized for the New Bottleneck: Memory Access
VLDB '99 Proceedings of the 25th International Conference on Very Large Data Bases
Performance in Practice of String Hashing Functions
Proceedings of the Fifth International Conference on Database Systems for Advanced Applications (DASFAA)
Efficient trie-based sorting of large sets of strings
ACSC '03 Proceedings of the 26th Australasian computer science conference - Volume 16
Compression techniques for fast external sorting
The VLDB Journal — The International Journal on Very Large Data Bases
| Hi-index | 0.00 |
Evaluating a query can involve manipulation of large volumes of temporary data. When the volume of data becomes too great, activities such as joins and sorting must use disk, and cost minimisation involves complex trade-offs. In this paper, we explore the effect of compression on the cost of external sorting. Reduction in the volume of data potentially allows costs to be reduced - through reductions in disk traffic and numbers of temporary files - but on-the-fly compression can be slow and many compression methods do not allow random access to individual records. We investigate a range of compression techniques for this problem, and develop successful methods based on common letter sequences. Our experiments show that, for a given memory limit, the overheads of compression outweigh the benefits for smaller data volumes, but for large files compression can yield substantial gains, of one-third of costs in the best case tested. Even when the data is stored uncompressed, our results show that incorporation of compression can significantly accelerate query processing.