Sizing sketches: a rank-based analysis for similarity search

Authors:
Zhe Wang;Wei Dong;William Josephson;Qin Lv;Moses Charikar;Kai Li
Affiliations:
Princeton University, Princeton, NJ;Princeton University, Princeton, NJ;Princeton University, Princeton, NJ;Princeton University, Princeton, NJ;Princeton University, Princeton, NJ;Princeton University, Princeton, NJ
Venue:
Proceedings of the 2007 ACM SIGMETRICS international conference on Measurement and modeling of computer systems
Year:
2007

Citing 21
Cited 5

K-d trees for semidynamic point sets

SCG '90 Proceedings of the sixth annual symposium on Computational geometry
Point location in arrangements of hyperplanes

Information and Computation
The SR-tree: an index structure for high-dimensional nearest neighbor queries

SIGMOD '97 Proceedings of the 1997 ACM SIGMOD international conference on Management of data
Approximate nearest neighbors: towards removing the curse of dimensionality

STOC '98 Proceedings of the thirtieth annual ACM symposium on Theory of computing
Unsupervised Segmentation of Color-Texture Regions in Images and Video

IEEE Transactions on Pattern Analysis and Machine Intelligence
Similarity estimation techniques from rounding algorithms

STOC '02 Proceedings of the thiry-fourth annual ACM symposium on Theory of computing
R-trees: a dynamic index structure for spatial searching

SIGMOD '84 Proceedings of the 1984 ACM SIGMOD international conference on Management of data
A Quantitative Analysis and Performance Study for Similarity-Search Methods in High-Dimensional Spaces

VLDB '98 Proceedings of the 24rd International Conference on Very Large Data Bases
Similarity Search in High Dimensions via Hashing

VLDB '99 Proceedings of the 25th International Conference on Very Large Data Bases
The A-tree: An Index Structure for High-Dimensional Spaces Using Relative Approximation

VLDB '00 Proceedings of the 26th International Conference on Very Large Data Bases
Rotation invariant spherical harmonic representation of 3D shape descriptors

Proceedings of the 2003 Eurographics/ACM SIGGRAPH symposium on Geometry processing
MARSYAS: a framework for audio analysis

Organised Sound
MARSYAS: a framework for audio analysis

Organised Sound
Navigating nets: simple algorithms for proximity search

SODA '04 Proceedings of the fifteenth annual ACM-SIAM symposium on Discrete algorithms
Locality-sensitive hashing scheme based on p-stable distributions

SCG '04 Proceedings of the twentieth annual symposium on Computational geometry
Image similarity search with compact data structures

Proceedings of the thirteenth ACM international conference on Information and knowledge management
The Active Vertice method: a performant filtering approach to high-dimensional indexing

Data & Knowledge Engineering
Entropy based nearest neighbor search in high dimensions

SODA '06 Proceedings of the seventeenth annual ACM-SIAM symposium on Discrete algorithm
Cover trees for nearest neighbor

ICML '06 Proceedings of the 23rd international conference on Machine learning
Efficient filtering with sketches in the ferret toolkit

MIR '06 Proceedings of the 8th ACM international workshop on Multimedia information retrieval
Ferret: a toolkit for content-based similarity search of feature-rich data

Proceedings of the 1st ACM SIGOPS/EuroSys European Conference on Computer Systems 2006

Embedding and similarity search for point sets under translation

Proceedings of the twenty-fourth annual symposium on Computational geometry
Asymmetric distance estimation with sketches for similarity search in high-dimensional spaces

Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval
Modeling LSH for performance tuning

Proceedings of the 17th ACM conference on Information and knowledge management
Efficient Similarity Search by Reducing I/O with Compressed Sketches

SISAP '09 Proceedings of the 2009 Second International Workshop on Similarity Search and Applications
Active multiple kernel learning for interactive 3D object retrieval systems

ACM Transactions on Interactive Intelligent Systems (TiiS)

Quantified Score

Hi-index	0.01

Visualization

Abstract

Sketches are compact data structures that can be used to estimate properties of the original data in building large-scale search engines and data analysis systems. Recent theoretical and experimental studies have shown that sketches constructed from feature vectors using randomized projections can effectively approximate L1 distance on the feature vectors with the Hamming distance on their sketches. Furthermore, such sketches can achieve good filtering accuracy while reducing the metadata space requirement and speeding up similarity searches by an order of magnitude. However, it is not clear how to choose the size of the sketches since it depends ondata type, dataset size, and desired filtering quality. In real systems designs, it is necessary to understand how to choose sketch size without the dataset, or at least without the whole datase. This paper presents an analytical model and experimental results to help system designers make such design decisions. We present arank-based filtering model that describes the relationship between sketch size and data set size based on the dataset distance distribution. Our experimental results with several datasets including images, audio, and 3D shapes show that the model yields good, conservative predictions. We show that the parameters of the model can be set with a small sample data set and the resulting model can make good predictions for a large dataset. We illustrate how to apply the approach with a concrete example.