A simple algorithm for finding frequent elements in streams and bags

Authors:
Richard M. Karp;Scott Shenker;Christos H. Papadimitriou
Affiliations:
International Computer Science Institute and University of California, Berkeley, California;International Computer Science Institute and University of California, Berkeley, California;University of California, Berkeley, California
Venue:
ACM Transactions on Database Systems (TODS)
Year:
2003

Citing 8
Cited 133

The space complexity of approximating the frequency moments

STOC '96 Proceedings of the twenty-eighth annual ACM symposium on Theory of computing
Bottom-up computation of sparse and Iceberg CUBE

SIGMOD '99 Proceedings of the 1999 ACM SIGMOD international conference on Management of data
Synopsis data structures for massive data sets

External memory algorithms
New directions in traffic measurement and accounting

IMW '01 Proceedings of the 1st ACM SIGCOMM Workshop on Internet Measurement
Approximate fairness through differential dropping: (summary)

ACM SIGCOMM Computer Communication Review
Computing Iceberg Queries Efficiently

VLDB '98 Proceedings of the 24rd International Conference on Very Large Data Bases
Finding Frequent Items in Data Streams

ICALP '02 Proceedings of the 29th International Colloquium on Automata, Languages and Programming
Frequency Estimation of Internet Packet Streams with Limited Space

ESA '02 Proceedings of the 10th Annual European Symposium on Algorithms

What's hot and what's not: tracking most frequent items dynamically

Proceedings of the twenty-second ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems
New directions in traffic measurement and accounting: Focusing on the elephants, ignoring the mice

ACM Transactions on Computer Systems (TOCS)
Space-code bloom filter for efficient traffic flow measurement

Proceedings of the 3rd ACM SIGCOMM conference on Internet measurement
Dynamically maintaining frequent items over a data stream

CIKM '03 Proceedings of the twelfth international conference on Information and knowledge management
Finding frequent items in data streams

Theoretical Computer Science - Special issue on automata, languages and programming
Deterministic sampling and range counting in geometric data streams

SCG '04 Proceedings of the twentieth annual symposium on Computational geometry
Holistic UDAFs at streaming speeds

SIGMOD '04 Proceedings of the 2004 ACM SIGMOD international conference on Management of data
On mining webclick streams for path traversal patterns

Proceedings of the 13th international World Wide Web conference on Alternate track papers & posters
Finding (Recently) Frequent Items in Distributed Data Streams

ICDE '05 Proceedings of the 21st International Conference on Data Engineering
Approximate counts and quantiles over sliding windows

PODS '04 Proceedings of the twenty-third ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems
Duplicate detection in click streams

WWW '05 Proceedings of the 14th international conference on World Wide Web
What's hot and what's not: tracking most frequent items dynamically

ACM Transactions on Database Systems (TODS) - Special Issue: SIGMOD/PODS 2003
Space complexity of hierarchical heavy hitters in multi-dimensional data streams

Proceedings of the twenty-fourth ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems
Fast and approximate stream mining of quantiles and frequencies using graphics processors

Proceedings of the 2005 ACM SIGMOD international conference on Management of data
Efficient evaluation of XQuery over streaming data

VLDB '05 Proceedings of the 31st international conference on Very large data bases
Extracting Frequent Subsequences from a Single Long Data Sequence: A Novel Anti-Monotonic Measure and a Simple On-Line Algorithm

ICDM '05 Proceedings of the Fifth IEEE International Conference on Data Mining
Maintaining significant stream statistics over sliding windows

SODA '06 Proceedings of the seventeenth annual ACM-SIAM symposium on Discrete algorithm
What's new: finding significant differences in network data streams

IEEE/ACM Transactions on Networking (TON)
Research issues in data stream association rule mining

ACM SIGMOD Record
A simpler and more efficient deterministic scheme for finding frequent items over sliding windows

Proceedings of the twenty-fifth ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems
Finding global icebergs over distributed data sets

Proceedings of the twenty-fifth ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems
DSM-PLW: single-pass mining of path traversal patterns over streaming web click-sequences

Computer Networks: The International Journal of Computer and Telecommunications Networking - Web dynamics
An integrated efficient solution for computing frequent and top-k elements in data streams

ACM Transactions on Database Systems (TODS)
Data streams: algorithms and applications

Foundations and Trends® in Theoretical Computer Science
Mining evolving data streams for frequent patterns

Pattern Recognition
Towards a new approach for mining frequent itemsets on data stream

Journal of Intelligent Information Systems
Association rules mining using heavy itemsets

Data & Knowledge Engineering
Deterministic sampling and range counting in geometric data streams

ACM Transactions on Algorithms (TALG)
Fast data stream algorithms using associative memories

Proceedings of the 2007 ACM SIGMOD international conference on Management of data
Joint data streaming and sampling techniques for detection of super sources and destinations

IMC '05 Proceedings of the 5th ACM SIGCOMM conference on Internet Measurement
Answering ad hoc aggregate queries from data streams using prefix aggregate trees

Knowledge and Information Systems
Finding hierarchical heavy hitters in data streams

VLDB '03 Proceedings of the 29th international conference on Very large data bases - Volume 29
Removing manually generated boilerplate from electronic texts: experiments with project Gutenberg e-books

CASCON '07 Proceedings of the 2007 conference of the center for advanced studies on Collaborative research
Finding hierarchical heavy hitters in streaming data

ACM Transactions on Knowledge Discovery from Data (TKDD)
Probabilistic lossy counting: an efficient algorithm for finding heavy hitters

ACM SIGCOMM Computer Communication Review
Hyperion: high volume stream archival for retrospective querying

ATC'07 2007 USENIX Annual Technical Conference on Proceedings of the USENIX Annual Technical Conference
Approximate mining of frequent patterns on streams

Intelligent Data Analysis - Knowlegde Discovery from Data Streams
Finding popular categories for RFID tags

Proceedings of the 9th ACM international symposium on Mobile ad hoc networking and computing
Finding frequent items in probabilistic data

Proceedings of the 2008 ACM SIGMOD international conference on Management of data
Mining frequent items in a stream using flexible windows

Intelligent Data Analysis - Knowledge Discovery from Data Streams
Computing Frequent Elements Using Gossip

SIROCCO '08 Proceedings of the 15th international colloquium on Structural Information and Communication Complexity
Finding Frequent Items over General Update Streams

SSDBM '08 Proceedings of the 20th international conference on Scientific and Statistical Database Management
Efficiently Discovering Recent Frequent Items in Data Streams

SSDBM '08 Proceedings of the 20th international conference on Scientific and Statistical Database Management
Finding Frequent Items in a Turnstile Data Stream

COCOON '08 Proceedings of the 14th annual international conference on Computing and Combinatorics
Efficient algorithms for stream mining of constrained frequent patterns in a limited memory environment

IDEAS '08 Proceedings of the 2008 international symposium on Database engineering & applications
Adaptive shared-state sampling

Proceedings of the 8th ACM SIGCOMM conference on Internet measurement
SLEUTH: Single-pubLisher attack dEtection Using correlaTion Hunting

Proceedings of the VLDB Endowment
Finding frequent items in data streams

Proceedings of the VLDB Endowment
ODMCA: An adaptive data mining control algorithm in multicarrier networks

Computer Communications
Mining frequent closed itemsets from a landmark window over online data streams

Computers & Mathematics with Applications
The design of a query monitoring system

ACM Transactions on Database Systems (TODS)
Semantics and implementation of continuous sliding window queries over data streams

ACM Transactions on Database Systems (TODS)
Frequent items in streaming data: An experimental evaluation of the state-of-the-art

Data & Knowledge Engineering
HIDS: a multifunctional generator of hierarchical data streams

ACM SIGMIS Database
A novel hash-based approach for mining frequent itemsets over data streams requiring less memory space

Data Mining and Knowledge Discovery
Space-optimal heavy hitters with strong error bounds

Proceedings of the twenty-eighth ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems
Optimal tracking of distributed heavy hitters and quantiles

Proceedings of the twenty-eighth ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems
Mining Sequential Patterns in Data Stream

ISNN 2009 Proceedings of the 6th International Symposium on Neural Networks: Advances in Neural Networks - Part II
Finding the frequent items in streams of data

Communications of the ACM - A View of Parallel Computing
Small synopses for group-by query verification on outsourced data streams

ACM Transactions on Database Systems (TODS)
Deterministically Estimating Data Stream Frequencies

COCOA '09 Proceedings of the 3rd International Conference on Combinatorial Optimization and Applications
The Frequent Items Problem, under Polynomial Decay, in the Streaming Model

SPIRE '09 Proceedings of the 16th International Symposium on String Processing and Information Retrieval
Which Is Better for Frequent Pattern Mining: Approximate Counting or Sampling?

DaWaK '09 Proceedings of the 11th International Conference on Data Warehousing and Knowledge Discovery
Online Evaluation of Patterns from Evolving Web Data Streams

WI-IAT '09 Proceedings of the 2009 IEEE/WIC/ACM International Joint Conference on Web Intelligence and Intelligent Agent Technology - Volume 01
Precomputing search features for fast and accurate query classification

Proceedings of the third ACM international conference on Web search and data mining
Methods for finding frequent items in data streams

The VLDB Journal — The International Journal on Very Large Data Bases
Finding frequent items over sliding windows with constant update time

Information Processing Letters
Online mining maximal frequent structures in continuous landmark melody streams

Pattern Recognition Letters
Learning When Concepts Abound

The Journal of Machine Learning Research
Discovering correlated items in data streams

PAKDD'07 Proceedings of the 11th Pacific-Asia conference on Advances in knowledge discovery and data mining
A robust approach to find effective items in distributed data streams

LSMS'07 Proceedings of the Life system modeling and simulation 2007 international conference on Bio-Inspired computational intelligence and applications
Finding frequent elements in non-bursty streams

ESA'07 Proceedings of the 15th annual European conference on Algorithms
Finding frequent items in data streams using ESBF

PAKDD'07 Proceedings of the 2007 international conference on Emerging technologies in knowledge discovery and data mining
CLAIM: an efficient method for relaxed frequent closed itemsets mining over stream data

DASFAA'07 Proceedings of the 12th international conference on Database systems for advanced applications
Aggregate computation over data streams

APWeb'08 Proceedings of the 10th Asia-Pacific web conference on Progress in WWW research and development
Finding heavy hitters over the sliding window of a weighted data stream

LATIN'08 Proceedings of the 8th Latin American conference on Theoretical informatics
Sequential patterns mining scaling with data stream based on LSP-tree

FSKD'09 Proceedings of the 6th international conference on Fuzzy systems and knowledge discovery - Volume 5
Lower bounds on frequency estimation of data streams

CSR'08 Proceedings of the 3rd international conference on Computer science: theory and applications
The frequent items problem, under polynomial decay, in the streaming model

Theoretical Computer Science
High-speed per-flow traffic measurement with probabilistic multiplicity counting

INFOCOM'10 Proceedings of the 29th conference on Information communications
Mining frequent patterns from network flows for monitoring network

Expert Systems with Applications: An International Journal
Mining discriminative items in multiple data streams

World Wide Web
Space-optimal heavy hitters with strong error bounds

ACM Transactions on Database Systems (TODS)
Identifying frequent items in a network using gossip

Journal of Parallel and Distributed Computing
An Ω(1/ε log 1/ε) space lower bound for finding ε-approximate quantiles in a data stream

FAW'10 Proceedings of the 4th international conference on Frontiers in algorithmics
Parallelizing weighted frequency counting in high-speed network monitoring

Computer Communications
Result enrichment in commerce search using browse trails

Proceedings of the fourth ACM international conference on Web search and data mining
Uncovering Global Icebergs in Distributed Streams: Results and Implications

Journal of Network and Systems Management
Pricing and unresponsive flows purging for global rate enhancement

Journal of Electrical and Computer Engineering
Discovery of frequent patterns in transactional data streams

Transactions on large-scale data- and knowledge-centered systems II
Discovery of frequent patterns in transactional data streams

Transactions on large-scale data- and knowledge-centered systems II
Mining frequent itemsets over distributed data streams by continuously maintaining a global synopsis

Data Mining and Knowledge Discovery
Streaming algorithms for 2-coloring uniform hypergraphs

WADS'11 Proceedings of the 12th international conference on Algorithms and data structures
Finding frequent elements in compressed 2D arrays and strings

SPIRE'11 Proceedings of the 18th international conference on String processing and information retrieval
Processing frequent items over distributed data streams

APWeb'05 Proceedings of the 7th Asia-Pacific web conference on Web Technologies Research and Development
Code transformations for one-pass analysis

LCPC'05 Proceedings of the 18th international conference on Languages and Compilers for Parallel Computing
Hardware enhanced mining for association rules

PAKDD'06 Proceedings of the 10th Pacific-Asia conference on Advances in Knowledge Discovery and Data Mining
Dynamically mining frequent patterns over online data streams

ISPA'05 Proceedings of the Third international conference on Parallel and Distributed Processing and Applications
Finding frequent patterns in a string in sublinear time

ESA'05 Proceedings of the 13th annual European conference on Algorithms
A new algorithm for long flows Statistics—MGCBF

ICCS'06 Proceedings of the 6th international conference on Computational Science - Volume Part IV
A single-pass online data mining algorithm combined with control theory with limited memory in dynamic data streams

GCC'05 Proceedings of the 4th international conference on Grid and Cooperative Computing
Efficient computation of frequent and top-k elements in data streams

ICDT'05 Proceedings of the 10th international conference on Database Theory
Adaptive spatial partitioning for multidimensional data streams

ISAAC'04 Proceedings of the 15th international conference on Algorithms and Computation
False-Negative frequent items mining from data streams with bursting

DASFAA'05 Proceedings of the 10th international conference on Database Systems for Advanced Applications
Approximating frequent items in asynchronous data stream over a sliding window

WAOA'09 Proceedings of the 7th international conference on Approximation and Online Algorithms
Dynamic range majority data structures

ISAAC'11 Proceedings of the 22nd international conference on Algorithms and Computation
A randomized algorithm for finding frequent elements in streams using o(loglogn) space

ISAAC'11 Proceedings of the 22nd international conference on Algorithms and Computation
A sliding window-based false-negative approach for ubiquitous data stream analysis

International Journal of Communication Systems
Detecting anomalies in backbone network traffic: a performance comparison among several change detection methods

International Journal of Sensor Networks
Sketching and streaming algorithms for processing massive data

XRDS: Crossroads, The ACM Magazine for Students - Big Data
Differentially private continual monitoring of heavy hitters from distributed streams

PETS'12 Proceedings of the 12th international conference on Privacy Enhancing Technologies
Sequential patterns mining on high-dimensional data stream

ICIC'12 Proceedings of the 8th international conference on Intelligent Computing Theories and Applications
DWFIST: leveraging calendar-based pattern mining in data streams

DaWaK'07 Proceedings of the 9th international conference on Data Warehousing and Knowledge Discovery
CR-PRECIS: a deterministic summary structure for update data streams

ESCAPE'07 Proceedings of the First international conference on Combinatorics, Algorithms, Probabilistic and Experimental Methodologies
Improved counter based algorithms for frequent pairs mining in transactional data streams

ECML PKDD'12 Proceedings of the 2012 European conference on Machine Learning and Knowledge Discovery in Databases - Volume Part I
Size matters: finding the most informative set of window lengths

ECML PKDD'12 Proceedings of the 2012 European conference on Machine Learning and Knowledge Discovery in Databases - Volume Part II
Per-flow traffic measurement through randomized counter sharing

IEEE/ACM Transactions on Networking (TON)
Optimizing adaptive multi-route query processing via time-partitioned indices

Journal of Computer and System Sciences
On the streaming complexity of computing local clustering coefficients

Proceedings of the sixth ACM international conference on Web search and data mining
Scalable identification and measurement of heavy-hitters

Computer Communications
ProFID: Practical frequent items discovery in peer-to-peer networks

Future Generation Computer Systems
Online identification of frequently executed acyclic paths by leveraging data stream algorithms

Proceedings of the 28th Annual ACM Symposium on Applied Computing
High throughput heavy hitter aggregation for modern SIMD processors

Proceedings of the Ninth International Workshop on Data Management on New Hardware
Simple and deterministic matrix sketching

Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining
Sketching for big data recommender systems using fast pseudo-random fingerprints

ICALP'13 Proceedings of the 40th international conference on Automata, Languages, and Programming - Volume Part II
Better space bounds for parameterized range majority and minority

WADS'13 Proceedings of the 13th international conference on Algorithms and Data Structures
Identifying streaming frequent items in ad hoc time windows

Data & Knowledge Engineering
A methodological overview on anomaly detection

DataTraffic Monitoring and Analysis
Stream mining on univariate uncertain data

Applied Intelligence
Mining frequent itemsets in a stream

Information Systems
Mining frequent items in data stream using time fading model

Information Sciences: an International Journal
Efficient frequent itemset mining methods over time-sensitive streams

Knowledge-Based Systems
Mining top-k frequent patterns over data streams sliding window

Journal of Intelligent Information Systems

Quantified Score

Hi-index	0.00

Visualization

Abstract

We present a simple, exact algorithm for identifying in a multiset the items with frequency more than a threshold θ. The algorithm requires two passes, linear time, and space 1/θ. The first pass is an on-line algorithm, generalizing a well-known algorithm for finding a majority element, for identifying a set of at most 1/θ items that includes, possibly among others, all items with frequency greater than θ.