Optimal approximations of the frequency moments of data streams

Authors:
Piotr Indyk;David Woodruff
Affiliations:
Massachusetts Institute of Technology, Cambridge, MA;Massachusetts Institute of Technology, Cambridge, MA
Venue:
Proceedings of the thirty-seventh annual ACM symposium on Theory of computing
Year:
2005

Citing 10
Cited 50

The space complexity of approximating the frequency moments

STOC '96 Proceedings of the twenty-eighth annual ACM symposium on Theory of computing
A small approximately min-wise independent family of hash functions

Journal of Algorithms
Space lower bounds for distance approximation in the data stream model

STOC '02 Proceedings of the thiry-fourth annual ACM symposium on Theory of computing
An Approximate L1-Difference Algorithm for Massive Data Streams

SIAM Journal on Computing
An Information Statistics Approach to Data Stream and Communication Complexity

FOCS '02 Proceedings of the 43rd Symposium on Foundations of Computer Science
Counting Distinct Elements in a Data Stream

RANDOM '02 Proceedings of the 6th International Workshop on Randomization and Approximation Techniques
An improved data stream algorithm for frequency moments

SODA '04 Proceedings of the fifteenth annual ACM-SIAM symposium on Discrete algorithms
Tabulation based 4-universal hashing with applications to second moment estimation

SODA '04 Proceedings of the fifteenth annual ACM-SIAM symposium on Discrete algorithms
Finding frequent items in data streams

Theoretical Computer Science - Special issue on automata, languages and programming
Stable distributions, pseudorandom generators, embeddings, and data stream computation

Journal of the ACM (JACM)

Simpler algorithm for estimating frequency moments of data streams

SODA '06 Proceedings of the seventeenth annual ACM-SIAM symposium on Discrete algorithm
The space complexity of pass-efficient algorithms for clustering

SODA '06 Proceedings of the seventeenth annual ACM-SIAM symposium on Discrete algorithm
Stable distributions, pseudorandom generators, embeddings, and data stream computation

Journal of the ACM (JACM)
Data streams: algorithms and applications

Foundations and Trends® in Theoretical Computer Science
Estimating entropy over data streams

ESA'06 Proceedings of the 14th conference on Annual European Symposium - Volume 14
A near-optimal algorithm for computing the entropy of a stream

SODA '07 Proceedings of the eighteenth annual ACM-SIAM symposium on Discrete algorithms
Declaring independence via the sketching of sketches

Proceedings of the nineteenth annual ACM-SIAM symposium on Discrete algorithms
Robust lower bounds for communication and stream computation

STOC '08 Proceedings of the fortieth annual ACM symposium on Theory of computing
Sketching information divergences

Machine Learning
Estimating Hybrid Frequency Moments of Data Streams

FAW '08 Proceedings of the 2nd annual international workshop on Frontiers in Algorithmics
On Estimating Frequency Moments of Data Streams

APPROX '07/RANDOM '07 Proceedings of the 10th International Workshop on Approximation and the 11th International Workshop on Randomization, and Combinatorial Optimization. Algorithms and Techniques
On Finding Frequent Elements in a Data Stream

APPROX '07/RANDOM '07 Proceedings of the 10th International Workshop on Approximation and the 11th International Workshop on Randomization, and Combinatorial Optimization. Algorithms and Techniques
Compressed counting

SODA '09 Proceedings of the twentieth Annual ACM-SIAM Symposium on Discrete Algorithms
Overcoming the l1 non-embeddability barrier: algorithms for product metrics

SODA '09 Proceedings of the twentieth Annual ACM-SIAM Symposium on Discrete Algorithms
Polylog Space Compression Is Incomparable with Lempel-Ziv and Pushdown Compression

SOFSEM '09 Proceedings of the 35th Conference on Current Trends in Theory and Practice of Computer Science
Optimal sampling from sliding windows

Proceedings of the twenty-eighth ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems
Small synopses for group-by query verification on outsourced data streams

ACM Transactions on Database Systems (TODS)
A Note on Estimating Hybrid Frequency Moment of Data Streams

AAIM '09 Proceedings of the 5th International Conference on Algorithmic Aspects in Information and Management
Deterministically Estimating Data Stream Frequencies

COCOA '09 Proceedings of the 3rd International Conference on Combinatorial Optimization and Applications
Revisiting the Direct Sum Theorem and Space Lower Bounds in Random Order Streams

ICALP '09 Proceedings of the 36th International Colloquium on Automata, Languages and Programming: Part I
Hellinger Strikes Back: A Note on the Multi-party Information Complexity of AND

APPROX '09 / RANDOM '09 Proceedings of the 12th International Workshop and 13th International Workshop on Approximation, Randomization, and Combinatorial Optimization. Algorithms and Techniques
Sketching information divergences

COLT'07 Proceedings of the 20th annual conference on Learning theory
Finding frequent elements in non-bursty streams

ESA'07 Proceedings of the 15th annual European conference on Algorithms
Aggregate computation over data streams

APWeb'08 Proceedings of the 10th Asia-Pacific web conference on Progress in WWW research and development
Improving compressed counting

UAI '09 Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence
A near-optimal algorithm for estimating the entropy of a stream

ACM Transactions on Algorithms (TALG)
Measuring independence of datasets

Proceedings of the forty-second ACM symposium on Theory of computing
Zero-one frequency laws

Proceedings of the forty-second ACM symposium on Theory of computing
Information complexity: a tutorial

Proceedings of the twenty-ninth ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems
1-pass relative-error Lp-sampling with applications

SODA '10 Proceedings of the twenty-first annual ACM-SIAM symposium on Discrete Algorithms
Periodicity in streams

APPROX/RANDOM'10 Proceedings of the 13th international conference on Approximation, and 14 the International conference on Randomization, and combinatorial optimization: algorithms and techniques
Effective Computations on Sliding Windows

SIAM Journal on Computing
Polylogarithmic approximation for edit distance and the asymmetric query complexity

Property testing
Polylogarithmic approximation for edit distance and the asymmetric query complexity

Property testing
Finding heavy distinct hitters in data streams

Proceedings of the twenty-third annual ACM symposium on Parallelism in algorithms and architectures
Near-optimal private approximation protocols via a black box transformation

Proceedings of the forty-third annual ACM symposium on Theory of computing
Fast moment estimation in data streams in optimal space

Proceedings of the forty-third annual ACM symposium on Theory of computing
Subspace embeddings for the L1-norm with applications

Proceedings of the forty-third annual ACM symposium on Theory of computing
Optimal sampling from sliding windows

Journal of Computer and System Sciences
The Value of Multiple Read/Write Streams for Approximating Frequency Moments

ACM Transactions on Computation Theory (TOCT)
Estimating entropy and entropy norm on data streams

STACS'06 Proceedings of the 23rd Annual conference on Theoretical Aspects of Computer Science
Estimating hybrid frequency moments of data streams

Journal of Combinatorial Optimization
Graph sketches: sparsification, spanners, and subgraphs

PODS '12 Proceedings of the 31st symposium on Principles of Database Systems
Space-efficient estimation of statistics over sub-sampled streams

PODS '12 Proceedings of the 31st symposium on Principles of Database Systems
Rectangle-efficient aggregation in spatial data streams

PODS '12 Proceedings of the 31st symposium on Principles of Database Systems
Tight bounds for distributed functional monitoring

STOC '12 Proceedings of the forty-fourth annual ACM symposium on Theory of computing
Statistical estimation with bounded memory

Statistics and Computing
Synopses for Massive Data: Samples, Histograms, Wavelets, Sketches

Foundations and Trends in Databases
Streaming algorithms measured in terms of the computed quantity

COCOON'07 Proceedings of the 13th annual international conference on Computing and Combinatorics
Tight lower bound for linear sketches of moments

ICALP'13 Proceedings of the 40th international conference on Automata, Languages, and Programming - Volume Part I

Quantified Score

Hi-index	0.00

Visualization

Abstract

We give a 1-pass Õ(m1-2⁄k)-space algorithm for computing the k-th frequency moment of a data stream for any real k 2. Together with the lower bounds of [1, 2, 4], this resolves the main problem left open by Alon et al in 1996 [1]. Our algorithm also works for streams with deletions and thus gives an Õ(m 1-2⁄p) space algorithm for the Lp difference problem for any p 2. This essentially matches the known Ω(m1-2⁄p-o(1)) lower bound of [12, 2]. Finally the update time of our algorithms is Õ(1).