Inverted Index Compression Using Word-Aligned Binary Codes

Authors:
Vo Ngoc Anh;Alistair Moffat
Affiliations:
Department of Computer Science and Software Engineering, The University of Melbourne, Victoria 3010, Australia;Department of Computer Science and Software Engineering, The University of Melbourne, Victoria 3010, Australia. alistair@cs.mu.oz.au
Venue:
Information Retrieval
Year:
2005

Citing 16
Cited 56

Automatic text processing: the transformation, analysis, and retrieval of information by computer

Automatic text processing: the transformation, analysis, and retrieval of information by computer
Information retrieval: data structures and algorithms

Information retrieval: data structures and algorithms
Overview of the second text retrieval conference (TREC-2)

TREC-2 Proceedings of the second conference on Text retrieval conference
Adding compression to a full-text retrieval system

Software—Practice & Experience
Filtered document retrieval with frequency-sorted indexes

Journal of the American Society for Information Science
Managing gigabytes (2nd ed.): compressing and indexing documents and images

Managing gigabytes (2nd ed.): compressing and indexing documents and images
Fast and flexible word searching on compressed text

ACM Transactions on Information Systems (TOIS)
Vector-space ranking with effective early termination

Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval
Modern Information Retrieval

Modern Information Retrieval
Compression of inverted indexes For fast query evaluation

SIGIR '02 Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval
Does WT10g look like the web?

SIGIR '02 Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval
Binary Interpolative Coding for Effective Index Compression

Information Retrieval
Compressing Inverted Files

Information Retrieval
Index Compression through Document Reordering

DCC '02 Proceedings of the Data Compression Conference
Engineering a multi-purpose test collection for web retrieval experiments

Information Processing and Management: an International Journal
Index compression using fixed binary codewords

ADC '04 Proceedings of the 15th Australasian database conference - Volume 27

Assigning identifiers to documents to enhance the clustering property of fulltext indexes

Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval
Index compression using fixed binary codewords

ADC '04 Proceedings of the 15th Australasian database conference - Volume 27
Simplified similarity scoring using term ranks

Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval
Fast on-line index construction by geometric partitioning

Proceedings of the 14th ACM international conference on Information and knowledge management
Unique-order interpolative coding for fast querying and space-efficient indexing in information retrieval systems

Information Processing and Management: an International Journal
Improved Word-Aligned Binary Compression for Text Indexing

IEEE Transactions on Knowledge and Data Engineering
Inverted files for text search engines

ACM Computing Surveys (CSUR)
Efficient online index maintenance for contiguous inverted lists

Information Processing and Management: an International Journal
Type less, find more: fast autocompletion search with a succinct index

SIGIR '06 Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval
Binary codes for locally homogeneous sequences

Information Processing Letters
Index compression is good, especially for random access

Proceedings of the sixteenth ACM conference on Conference on information and knowledge management
Sigma encoded inverted files

Proceedings of the sixteenth ACM conference on Conference on information and knowledge management
Hybrid index maintenance for contiguous inverted lists

Information Retrieval
Incremental cluster-based retrieval using compressed cluster-skipping inverted files

ACM Transactions on Information Systems (TOIS)
TinyLex: static n-gram index pruning with perfect recall

Proceedings of the 17th ACM conference on Information and knowledge management
Inverted index compression and query processing with optimized document ordering

Proceedings of the 18th international conference on World wide web
Reducing Space Requirements for Disk Resident Suffix Arrays

DASFAA '09 Proceedings of the 14th International Conference on Database Systems for Advanced Applications
Compressing term positions in web indexes

Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval
Space-economical partial gram indices for exact substring matching

Proceedings of the 18th ACM conference on Information and knowledge management
Unique-order interpolative coding for fast querying and space-efficient indexing in information retrieval systems

Information Processing and Management: an International Journal
Index compression using 64-bit words

Software—Practice & Experience
Sorting out the document identifier assignment problem

ECIR'07 Proceedings of the 29th European conference on IR research
Scalable techniques for document identifier assignment in inverted indexes

Proceedings of the 19th international conference on World wide web
Scalable online index construction with multi-core CPUs

ADC '10 Proceedings of the Twenty-First Australasian Conference on Database Technologies - Volume 104
VSEncoding: efficient coding and fast decoding of integer lists via dynamic programming

CIKM '10 Proceedings of the 19th ACM international conference on Information and knowledge management
Engineering basic algorithms of an in-memory text search engine

ACM Transactions on Information Systems (TOIS)
Fast and effective focused retrieval

INEX'09 Proceedings of the Focused retrieval and evaluation, and 8th international conference on Initiative for the evaluation of XML retrieval
NET-FLi: on-the-fly compression, archiving and indexing of streaming network traffic

Proceedings of the VLDB Endowment
Dual-sorted inverted lists

SPIRE'10 Proceedings of the 17th international conference on String processing and information retrieval
Inverted index compression via online document routing

Proceedings of the 20th international conference on World wide web
Reordering columns for smaller indexes

Information Sciences: an International Journal
Within-document term-based index pruning with statistical hypothesis testing

ECIR'11 Proceedings of the 33rd European conference on Advances in information retrieval
Efficient parallel lists intersection and index compression algorithms using graphics processing units

Proceedings of the VLDB Endowment
Interpolative coding of integer sequences supporting log-time random access

Information Processing and Management: an International Journal
University of Otago at INEX 2010

INEX'10 Proceedings of the 9th international conference on Initiative for the evaluation of XML retrieval: comparative evaluation of focused retrieval
SIMD-based decoding of posting lists

Proceedings of the 20th ACM international conference on Information and knowledge management
Indexes for highly repetitive document collections

Proceedings of the 20th ACM international conference on Information and knowledge management
Efficient phrase querying with flat position index

Proceedings of the 20th ACM international conference on Information and knowledge management
Relative Lempel-Ziv factorization for efficient storage and retrieval of web collections

Proceedings of the VLDB Endowment
Inverted files versus suffix arrays for locating patterns in primary memory

SPIRE'06 Proceedings of the 13th international conference on String Processing and Information Retrieval
Searching web data: An entity retrieval and high-performance indexing model

Web Semantics: Science, Services and Agents on the World Wide Web
Query retrieval enhancement based on Huffman index terms encoding

Proceedings of the 3rd International Conference on Information and Communication Systems
tsdb: a compressed database for time series

TMA'12 Proceedings of the 4th international conference on Traffic Monitoring and Analysis
Real-time creation of bitmap indexes on streaming network data

The VLDB Journal — The International Journal on Very Large Data Bases
To index or not to index: time-space trade-offs in search engines with positional ranking functions

SIGIR '12 Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval
Data-Parallel Decompression of Triangle Mesh Topology

Computer Graphics Forum
DACs: Bringing direct access to variable-length codes

Information Processing and Management: an International Journal
Dual-Sorted inverted lists in practice

SPIRE'12 Proceedings of the 19th international conference on String Processing and Information Retrieval
Implicit indexing of natural language text by reorganizing bytecodes

Information Retrieval
Quasi-succinct indices

Proceedings of the sixth ACM international conference on Web search and data mining
Document identifier reassignment and run-length-compressed inverted indexes for improved search performance

Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval
Faster and smaller inverted indices with treaps

Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval
Bitlist: new full-text index for low space cost and efficient keyword search

Proceedings of the VLDB Endowment
Managing short postings lists

Proceedings of the 18th Australasian Document Computing Symposium
Re-Ordered FEGC and Block Based FEGC for Inverted File Compression

International Journal of Information Retrieval Research
On the compression of search trees

Information Processing and Management: an International Journal

Quantified Score

Hi-index	0.00

Visualization

Abstract

We examine index representation techniques for document-based inverted files, and present a mechanism for compressing them using word-aligned binary codes. The new approach allows extremely fast decoding of inverted lists during query processing, while providing compression rates better than other high-throughput representations. Results are given for several large text collections in support of these claims, both for compression effectiveness and query efficiency.