Fast moment estimation in data streams in optimal space

Authors:
Daniel M. Kane;Jelani Nelson;Ely Porat;David P. Woodruff
Affiliations:
Harvard University, Cambridge, MA, USA;MIT, Cambridge, MA, USA;Bar-Ilan University, Ramat-Gan, Israel;IBM Research-Almaden, San Jose, CA, USA
Venue:
Proceedings of the forty-third annual ACM symposium on Theory of computing
Year:
2011

Citing 41
Cited 10

Storing a Sparse Table with 0(1) Worst Case Access Time

Journal of the ACM (JACM)
Modern computer algebra

Modern computer algebra
The space complexity of approximating the frequency moments

Journal of Computer and System Sciences
Sparse Representations for Image Decompositions

International Journal of Computer Vision
Space lower bounds for distance approximation in the data stream model

STOC '02 Proceedings of the thiry-fourth annual ACM symposium on Theory of computing
An Approximate L1-Difference Algorithm for Massive Data Streams

SIAM Journal on Computing
On the Surprising Behavior of Distance Metrics in High Dimensional Spaces

ICDT '01 Proceedings of the 8th International Conference on Database Theory
Finding Frequent Items in Data Streams

ICALP '02 Proceedings of the 29th International Colloquium on Automata, Languages and Programming
Sketch-based change detection: methods, evaluation, and applications

Proceedings of the 3rd ACM SIGCOMM conference on Internet measurement
Optimal space lower bounds for all frequency moments

SODA '04 Proceedings of the fifteenth annual ACM-SIAM symposium on Discrete algorithms
Tabulation based 4-universal hashing with applications to second moment estimation

SODA '04 Proceedings of the fifteenth annual ACM-SIAM symposium on Discrete algorithms
An information statistics approach to data stream and communication complexity

Journal of Computer and System Sciences - Special issue on FOCS 2002
Optimal approximations of the frequency moments of data streams

Proceedings of the thirty-seventh annual ACM symposium on Theory of computing
An improved data stream summary: the count-min sketch and its applications

Journal of Algorithms
Summarizing and mining inverse distributions on data streams via dynamic inverse sampling

VLDB '05 Proceedings of the 31st international conference on Very large data bases
Simpler algorithm for estimating frequency moments of data streams

SODA '06 Proceedings of the seventeenth annual ACM-SIAM symposium on Discrete algorithm
Streaming and sublinear approximation of entropy and information distances

SODA '06 Proceedings of the seventeenth annual ACM-SIAM symposium on Discrete algorithm
The complexity of massive data set computations

The complexity of massive data set computations
Stable distributions, pseudorandom generators, embeddings, and data stream computation

Journal of the ACM (JACM)
Data streams: algorithms and applications

Foundations and Trends® in Theoretical Computer Science
Improved Approximation Algorithms for Large Matrices via Random Projections

FOCS '06 Proceedings of the 47th Annual IEEE Symposium on Foundations of Computer Science
Detecting anomalies in network traffic using maximum entropy estimation

IMC '05 Proceedings of the 5th ACM SIGCOMM conference on Internet Measurement
Estimating entropy over data streams

ESA'06 Proceedings of the 14th conference on Annual European Symposium - Volume 14
A data streaming algorithm for estimating entropies of od flows

Proceedings of the 7th ACM SIGCOMM conference on Internet measurement
Estimators and tail bounds for dimension reduction in lα (0

Proceedings of the nineteenth annual ACM-SIAM symposium on Discrete algorithms
Declaring independence via the sketching of sketches

Proceedings of the nineteenth annual ACM-SIAM symposium on Discrete algorithms
Efficient and private distance approximation in the communication and streaming models

Efficient and private distance approximation in the communication and streaming models
Sketching information divergences

Machine Learning
Uniform Hashing in Constant Time and Optimal Space

SIAM Journal on Computing
Finding Frequent Items over General Update Streams

SSDBM '08 Proceedings of the 20th international conference on Scientific and Statistical Database Management
Sketching and Streaming Entropy via Approximation Theory

FOCS '08 Proceedings of the 2008 49th Annual IEEE Symposium on Foundations of Computer Science
Numerical linear algebra in the streaming model

Proceedings of the forty-first annual ACM symposium on Theory of computing
Hellinger Strikes Back: A Note on the Multi-party Information Complexity of AND

APPROX '09 / RANDOM '09 Proceedings of the 12th International Workshop and 13th International Workshop on Approximation, Randomization, and Combinatorial Optimization. Algorithms and Techniques
A near-optimal algorithm for estimating the entropy of a stream

ACM Transactions on Algorithms (TALG)
Fast Manhattan sketches in data streams

Proceedings of the twenty-ninth ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems
Information theory for data management

Proceedings of the 2010 ACM SIGMOD International Conference on Management of data
On the exact space complexity of sketching and streaming small norms

SODA '10 Proceedings of the twenty-first annual ACM-SIAM symposium on Discrete Algorithms
Lower bounds for sparse recovery

SODA '10 Proceedings of the twenty-first annual ACM-SIAM symposium on Discrete Algorithms
Bounded Independence Fools Degree-2 Threshold Functions

FOCS '10 Proceedings of the 2010 IEEE 51st Annual Symposium on Foundations of Computer Science
Estimating entropy and entropy norm on data streams

STACS'06 Proceedings of the 23rd Annual conference on Theoretical Aspects of Computer Science
Optimal bounds for Johnson-Lindenstrauss transforms and streaming problems with sub-constant error

Proceedings of the twenty-second annual ACM-SIAM symposium on Discrete Algorithms

Near-optimal private approximation protocols via a black box transformation

Proceedings of the forty-third annual ACM symposium on Theory of computing
Sparser Johnson-Lindenstrauss transforms

Proceedings of the twenty-third annual ACM-SIAM symposium on Discrete Algorithms
Graph sketches: sparsification, spanners, and subgraphs

PODS '12 Proceedings of the 31st symposium on Principles of Database Systems
Tight bounds for distributed functional monitoring

STOC '12 Proceedings of the forty-fourth annual ACM symposium on Theory of computing
Low rank approximation and regression in input sparsity time

Proceedings of the forty-fifth annual ACM symposium on Theory of computing
Homomorphic fingerprints under misalignments: sketching edit and shift distances

Proceedings of the forty-fifth annual ACM symposium on Theory of computing
Mergeable summaries

ACM Transactions on Database Systems (TODS) - Invited papers issue
Tight lower bound for linear sketches of moments

ICALP'13 Proceedings of the 40th international conference on Automata, Languages, and Programming - Volume Part I
Sketching for big data recommender systems using fast pseudo-random fingerprints

ICALP'13 Proceedings of the 40th international conference on Automata, Languages, and Programming - Volume Part II
Sparser Johnson-Lindenstrauss Transforms

Journal of the ACM (JACM)

Quantified Score

Hi-index	0.00

Visualization

Abstract

We give a space-optimal streaming algorithm with update time O(log2(1/ε)loglog(1/ε)) for approximating the pth frequency moment, 0 2). When combined with the work of [Harvey-Nelson-Onak, FOCS 2008], we also obtain the first algorithm for entropy estimation in turnstile streams which simultaneously achieves near-optimal space and fast update time.