Use of permutation prefixes for efficient and scalable approximate similarity search

  • Authors:
  • Andrea Esuli

  • Affiliations:
  • Istituto di Scienza e Tecnologie dell'Informazione, Consiglio Nazionale delle Ricerche, via Giuseppe Moruzzi, 1, 56124 Pisa, Italy

  • Venue:
  • Information Processing and Management: an International Journal
  • Year:
  • 2012

Quantified Score

Hi-index 0.00

Visualization

Abstract

We present the Permutation Prefix Index (this work is a revised and extended version of Esuli (2009b), presented at the 2009 LSDS-IR Workshop, held in Boston) (PP-Index), an index data structure that supports efficient approximate similarity search. The PP-Index belongs to the family of the permutation-based indexes, which are based on representing any indexed object with ''its view of the surrounding world'', i.e., a list of the elements of a set of reference objects sorted by their distance order with respect to the indexed object. In its basic formulation, the PP-Index is strongly biased toward efficiency. We show how the effectiveness can easily reach optimal levels just by adopting two ''boosting'' strategies: multiple index search and multiple query search, which both have nice parallelization properties. We study both the efficiency and the effectiveness properties of the PP-Index, experimenting with collections of sizes up to one hundred million objects, represented in a very high-dimensional similarity space.