The probabilistic communication complexity of set intersection
SIAM Journal on Discrete Mathematics
The space complexity of approximating the frequency moments
STOC '96 Proceedings of the twenty-eighth annual ACM symposium on Theory of computing
Communication complexity
New directions in traffic measurement and accounting
Proceedings of the 2002 conference on Applications, technologies, architectures, and protocols for computer communications
A signal analysis of network traffic anomalies
Proceedings of the 2nd ACM SIGCOMM Workshop on Internet measurment
Experience in measuring backbone traffic variability: models, metrics, measurements and meaning
Proceedings of the 2nd ACM SIGCOMM Workshop on Internet measurment
Estimating flow distributions from sampled flow statistics
Proceedings of the 2003 conference on Applications, technologies, architectures, and protocols for computer communications
Proceedings of the 3rd ACM SIGCOMM conference on Internet measurement
ACCEL-RATE: a faster mechanism for memory efficient per-flow traffic estimation
Proceedings of the joint international conference on Measurement and modeling of computer systems
Data streaming algorithms for efficient and accurate estimation of flow size distribution
Proceedings of the joint international conference on Measurement and modeling of computer systems
Diagnosing network-wide traffic anomalies
Proceedings of the 2004 conference on Applications, technologies, architectures, and protocols for computer communications
Online identification of hierarchical heavy hitters: algorithms, evaluation, and applications
Proceedings of the 4th ACM SIGCOMM conference on Internet measurement
On the difficulty of scalably detecting network attacks
Proceedings of the 11th ACM conference on Computer and communications security
Aberrant Behavior Detection in Time Series for Network Monitoring
LISA '00 Proceedings of the 14th USENIX conference on System administration
A data streaming algorithm for estimating subpopulation flow size distribution
SIGMETRICS '05 Proceedings of the 2005 ACM SIGMETRICS international conference on Measurement and modeling of computer systems
Profiling internet backbone traffic: behavior models and applications
Proceedings of the 2005 conference on Applications, technologies, architectures, and protocols for computer communications
Mining anomalies using traffic feature distributions
Proceedings of the 2005 conference on Applications, technologies, architectures, and protocols for computer communications
Detecting malicious network traffic using inverse distributions of packet contents
Proceedings of the 2005 ACM SIGCOMM workshop on Mining network data
Entropy Based Worm and Anomaly Detection in Fast IP Networks
WETICE '05 Proceedings of the 14th IEEE International Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprise
Streaming and sublinear approximation of entropy and information distances
SODA '06 Proceedings of the seventeenth annual ACM-SIAM symposium on Discrete algorithm
IMC '05 Proceedings of the 5th ACM SIGCOMM conference on Internet Measurement
Estimating entropy and entropy norm on data streams
STACS'06 Proceedings of the 23rd Annual conference on Theoretical Aspects of Computer Science
A near-optimal algorithm for computing the entropy of a stream
SODA '07 Proceedings of the eighteenth annual ACM-SIAM symposium on Discrete algorithms
A data streaming algorithm for estimating entropies of od flows
Proceedings of the 7th ACM SIGCOMM conference on Internet measurement
CSAMP: a system for network-wide flow monitoring
NSDI'08 Proceedings of the 5th USENIX Symposium on Networked Systems Design and Implementation
Streaming Estimation of Information-Theoretic Metrics for Anomaly Detection (Extended Abstract)
RAID '08 Proceedings of the 11th international symposium on Recent Advances in Intrusion Detection
An empirical evaluation of entropy-based traffic anomaly detection
Proceedings of the 8th ACM SIGCOMM conference on Internet measurement
Proceedings of the 40th Conference on Winter Simulation
Every microsecond counts: tracking fine-grain latencies with a lossy difference aggregator
Proceedings of the ACM SIGCOMM 2009 conference on Data communication
Sublinear estimation of entropy and information distances
ACM Transactions on Algorithms (TALG)
Fast classification and estimation of internet traffic flows
PAM'07 Proceedings of the 8th international conference on Passive and active network measurement
A QoS controller for adaptive streaming of 3D triangular scenes
Edutainment'07 Proceedings of the 2nd international conference on Technologies for e-learning and digital entertainment
An efficient and reliable DDoS attack detection using a fast entropy computation method
ISCIT'09 Proceedings of the 9th international conference on Communications and information technologies
UAI '09 Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence
A near-optimal algorithm for estimating the entropy of a stream
ACM Transactions on Algorithms (TALG)
An online framework for catching top spreaders and scanners
Computer Networks: The International Journal of Computer and Telecommunications Networking
Proceedings of the forty-second ACM symposium on Theory of computing
Tracking long duration flows in network traffic
INFOCOM'10 Proceedings of the 29th conference on Information communications
Revisiting the case for a minimalist approach for network flow monitoring
IMC '10 Proceedings of the 10th ACM SIGCOMM conference on Internet measurement
Space-efficient tracking of persistent items in a massive data stream
Proceedings of the 5th ACM international conference on Distributed event-based system
Virtual indexing based methods for estimating node connection degrees
Computer Networks: The International Journal of Computer and Telecommunications Networking
Router support for fine-grained latency measurements
IEEE/ACM Transactions on Networking (TON)
Characterizing per-application network traffic using entropy
ACM Transactions on Modeling and Computer Simulation (TOMACS)
An information-theoretical approach to high-speed flow nature identification
IEEE/ACM Transactions on Networking (TON)
Fake View Analytics in Online Video Services
Proceedings of Network and Operating System Support on Digital Audio and Video Workshop
| Hi-index | 0.00 |
Using entropy of traffic distributions has been shown to aid a wide variety of network monitoring applications such as anomaly detection, clustering to reveal interesting patterns, and traffic classification. However, realizing this potential benefit in practice requires accurate algorithms that can operate on high-speed links, with low CPU and memory requirements. In this paper, we investigate the problem of estimating the entropy in a streaming computation model. We give lower bounds for this problem, showing that neither approximation nor randomization alone will let us compute the entropy efficiently. We present two algorithms for randomly approximating the entropy in a time and space efficient manner, applicable for use on very high speed (greater than OC-48) links. The first algorithm for entropy estimation is inspired by the structural similarity with the seminal work of Alon et al. for estimating frequency moments, and we provide strong theoretical guarantees on the error and resource usage. Our second algorithm utilizes the observation that the performance of the streaming algorithm can be enhanced by separating the high-frequency items (or elephants) from the low-frequency items (or mice). We evaluate our algorithms on traffic traces from different deployment scenarios.