A linear-time probabilistic counting algorithm for database applications
ACM Transactions on Database Systems (TODS)
Efficient Hardware Hashing Functions for High Performance Computers
IEEE Transactions on Computers
Summary cache: a scalable wide-area web cache sharing protocol
IEEE/ACM Transactions on Networking (TON)
Space/time trade-offs in hash coding with allowable errors
Communications of the ACM
Charging from sampled network usage
IMW '01 Proceedings of the 1st ACM SIGCOMM Workshop on Internet Measurement
Traffic matrix estimation: existing techniques and new directions
Proceedings of the 2002 conference on Applications, technologies, architectures, and protocols for computer communications
New directions in traffic measurement and accounting
Proceedings of the 2002 conference on Applications, technologies, architectures, and protocols for computer communications
Iterative Bayesian estimation of network traffic matrices in the case of bursty flows
Proceedings of the 2nd ACM SIGCOMM Workshop on Internet measurment
Properties and prediction of flow statistics from sampled packet streams
Proceedings of the 2nd ACM SIGCOMM Workshop on Internet measurment
Fast accurate computation of large-scale IP traffic matrices from link loads
SIGMETRICS '03 Proceedings of the 2003 ACM SIGMETRICS international conference on Measurement and modeling of computer systems
Efficient implementation of a statistics counter architecture
SIGMETRICS '03 Proceedings of the 2003 ACM SIGMETRICS international conference on Measurement and modeling of computer systems
An information-theoretic approach to traffic matrix estimation
Proceedings of the 2003 conference on Applications, technologies, architectures, and protocols for computer communications
Estimating flow distributions from sampled flow statistics
Proceedings of the 2003 conference on Applications, technologies, architectures, and protocols for computer communications
Gigascope: a stream database for network applications
Proceedings of the 2003 ACM SIGMOD international conference on Management of data
Bitmap algorithms for counting active flows on high speed links
Proceedings of the 3rd ACM SIGCOMM conference on Internet measurement
Proceedings of the 3rd ACM SIGCOMM conference on Internet measurement
Sketch-based change detection: methods, evaluation, and applications
Proceedings of the 3rd ACM SIGCOMM conference on Internet measurement
A data streaming algorithm for estimating subpopulation flow size distribution
SIGMETRICS '05 Proceedings of the 2005 ACM SIGMETRICS international conference on Measurement and modeling of computer systems
Data streaming algorithms for accurate and efficient measurement of traffic and flow matrices
SIGMETRICS '05 Proceedings of the 2005 ACM SIGMETRICS international conference on Measurement and modeling of computer systems
Fast and accurate traffic matrix measurement using adaptive cardinality counting
Proceedings of the 2005 ACM SIGCOMM workshop on Mining network data
Data streaming algorithms for estimating entropy of network traffic
SIGMETRICS '06/Performance '06 Proceedings of the joint international conference on Measurement and modeling of computer systems
Design of a novel statistics counter architecture with optimal space and time efficiency
SIGMETRICS '06/Performance '06 Proceedings of the joint international conference on Measurement and modeling of computer systems
Finding global icebergs over distributed data sets
Proceedings of the twenty-fifth ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems
Proceedings of the 6th ACM SIGCOMM conference on Internet measurement
Joint data streaming and sampling techniques for detection of super sources and destinations
IMC '05 Proceedings of the 5th ACM SIGCOMM conference on Internet Measurement
WormShield: Fast Worm Signature Generation with Distributed Fingerprint Aggregation
IEEE Transactions on Dependable and Secure Computing
Probabilistic lossy counting: an efficient algorithm for finding heavy hitters
ACM SIGCOMM Computer Communication Review
A router-based technique to mitigate reduction of quality (RoQ) attacks
Computer Networks: The International Journal of Computer and Telecommunications Networking
Processing top k queries from samples
CoNEXT '06 Proceedings of the 2006 ACM CoNEXT conference
Counter braids: a novel counter architecture for per-flow measurement
SIGMETRICS '08 Proceedings of the 2008 ACM SIGMETRICS international conference on Measurement and modeling of computer systems
Estimating cardinality distributions in network traffic: extended abstract
SIGMETRICS '08 Proceedings of the 2008 ACM SIGMETRICS international conference on Measurement and modeling of computer systems
CSAMP: a system for network-wide flow monitoring
NSDI'08 Proceedings of the 5th USENIX Symposium on Networked Systems Design and Implementation
Processing top-k queries from samples
Computer Networks: The International Journal of Computer and Telecommunications Networking
EF-Greedy: A Novel Garbage Collection Policy for Flash Memory Based Embedded Systems
ICCS '07 Proceedings of the 7th international conference on Computational Science, Part IV: ICCS 2007
Summarizing Two-Dimensional Data with Skyline-Based Statistical Descriptors
SSDBM '08 Proceedings of the 20th international conference on Scientific and Statistical Database Management
An empirical evaluation of entropy-based traffic anomaly detection
Proceedings of the 8th ACM SIGCOMM conference on Internet measurement
Fast monitoring of traffic subpopulations
Proceedings of the 8th ACM SIGCOMM conference on Internet measurement
A resource-minimalist flow size histogram estimator
Proceedings of the 8th ACM SIGCOMM conference on Internet measurement
The eternal sunshine of the sketch data structure
Computer Networks: The International Journal of Computer and Telecommunications Networking
BRICK: a novel exact active statistics counter architecture
Proceedings of the 4th ACM/IEEE Symposium on Architectures for Networking and Communications Systems
Resilient workload manager: taming bursty workload of scaling internet applications
ICAC-INDST '09 Proceedings of the 6th international conference industry session on Autonomic computing and communications industry session
Fast classification and estimation of internet traffic flows
PAM'07 Proceedings of the 8th international conference on Passive and active network measurement
A novel algorithm for estimating flow length distributions-LSM
NPC'07 Proceedings of the 2007 IFIP international conference on Network and parallel computing
An online framework for catching top spreaders and scanners
Computer Networks: The International Journal of Computer and Telecommunications Networking
A new data streaming method for locating hosts with large connection degree
GLOBECOM'09 Proceedings of the 28th IEEE conference on Global telecommunications
Coordinated sampling sans origin-destination identifiers: algorithms and analysis
COMSNETS'10 Proceedings of the 2nd international conference on COMmunication systems and NETworks
Computer Networks: The International Journal of Computer and Telecommunications Networking
Tracking long duration flows in network traffic
INFOCOM'10 Proceedings of the 29th conference on Information communications
Measurouting: a framework for routing assisted traffic monitoring
INFOCOM'10 Proceedings of the 29th conference on Information communications
Revisiting the case for a minimalist approach for network flow monitoring
IMC '10 Proceedings of the 10th ACM SIGCOMM conference on Internet measurement
Parallelizing weighted frequency counting in high-speed network monitoring
Computer Communications
A flexible architecture integrating monitoring and analytics for managing large-scale data centers
Proceedings of the 8th ACM international conference on Autonomic computing
BRICK: a novel exact active statistics counter architecture
IEEE/ACM Transactions on Networking (TON)
Towards a universal sketch for origin-destination network measurements
NPC'11 Proceedings of the 8th IFIP international conference on Network and parallel computing
LEISURE: A Framework for Load-Balanced Network-Wide Traffic Measurement
Proceedings of the 2011 ACM/IEEE Seventh Symposium on Architectures for Networking and Communications Systems
An algorithm for estimation of flow length distributions using heavy-tailed feature
ICCS'06 Proceedings of the 6th international conference on Computational Science - Volume Part IV
Fit a compact spread estimator in small high-speed memory
IEEE/ACM Transactions on Networking (TON)
MeasuRouting: a framework for routing assisted traffic monitoring
IEEE/ACM Transactions on Networking (TON)
Virtual indexing based methods for estimating node connection degrees
Computer Networks: The International Journal of Computer and Telecommunications Networking
Flow level detection and filtering of low-rate DDoS
Computer Networks: The International Journal of Computer and Telecommunications Networking
Per-flow traffic measurement through randomized counter sharing
IEEE/ACM Transactions on Networking (TON)
Software defined traffic measurement with OpenSketch
nsdi'13 Proceedings of the 10th USENIX conference on Networked Systems Design and Implementation
Resource/accuracy tradeoffs in software-defined measurement
Proceedings of the second ACM SIGCOMM workshop on Hot topics in software defined networking
Modeling residual-geometric flow sampling
IEEE/ACM Transactions on Networking (TON)
Line speed accurate superspreader identification using dynamic error compensation
Computer Communications
Hi-index | 0.00 |
Knowing the distribution of the sizes of traffic flows passing through a network link helps a network operator to characterize network resource usage, infer traffic demands, detect traffic anomalies, and accommodate new traffic demands through better traffic engineering. Previous work on estimating the flow size distribution has been focused on making inferences from sampled network traffic. Its accuracy is limited by the (typically) low sampling rate required to make the sampling operation affordable. In this paper we present a novel data streaming algorithm to provide much more accurate estimates of flow distribution, using a "lossy data structure" which consists of an array of counters fitted well into SRAM. For each incoming packet, our algorithm only needs to increment one underlying counter, making the algorithm fast enough even for 40 Gbps (OC-768) links. The data structure is lossy in the sense that sizes of multiple flows may collide into the same counter. Our algorithm uses Bayesian statistical methods such as Expectation Maximization to infer the most likely flow size distribution that results in the observed counter values after collision. Evaluations of this algorithm on large Internet traces obtained from several sources (including a tier-1 ISP) demonstrate that it has very high measurement accuracy (within 2%). Our algorithm not only dramatically improves the accuracy of flow distribution measurement, but also contributes to the field of data streaming by formalizing an existing methodology and applying it to the context of estimating the flow-distribution.