Reversible sketches: enabling monitoring and analysis over high-speed data streams

Authors:
Robert Schweller;Zhichun Li;Yan Chen;Yan Gao;Ashish Gupta;Yin Zhang;Peter A. Dinda;Ming-Yang Kao;Gokhan Memik
Affiliations:
Department of Electrical Engineering and Computer Science, Northwestern University, Evanston, IL;Department of Electrical Engineering and Computer Science, Northwestern University, Evanston, IL;Department of Electrical Engineering and Computer Science, Northwestern University, Evanston, IL;Department of Electrical Engineering and Computer Science, Northwestern University, Evanston, IL;Department of Electrical Engineering and Computer Science, Northwestern University, Evanston, IL;Department of Computer Science, University of Texas at Austin, TX;Department of Electrical Engineering and Computer Science, Northwestern University, Evanston, IL;Department of Electrical Engineering and Computer Science, Northwestern University, Evanston, IL;Department of Electrical Engineering and Computer Science, Northwestern University, Evanston, IL
Venue:
IEEE/ACM Transactions on Networking (TON)
Year:
2007

Citing 20
Cited 13

Probabilistic counting algorithms for data base applications

Journal of Computer and System Sciences
Bro: a system for detecting network intruders in real-time

Computer Networks: The International Journal of Computer and Telecommunications Networking
Fast, small-space algorithms for approximate histogram maintenance

STOC '02 Proceedings of the thiry-fourth annual ACM symposium on Theory of computing
Practical automated detection of stealthy portscans

Journal of Computer Security
New directions in traffic measurement and accounting

Proceedings of the 2002 conference on Applications, technologies, architectures, and protocols for computer communications
A signal analysis of network traffic anomalies

Proceedings of the 2nd ACM SIGCOMM Workshop on Internet measurment
Properties and prediction of flow statistics from sampled packet streams

Proceedings of the 2nd ACM SIGCOMM Workshop on Internet measurment
Data streams: algorithms and applications

SODA '03 Proceedings of the fourteenth annual ACM-SIAM symposium on Discrete algorithms
Internet intrusions: global characteristics and prevalence

SIGMETRICS '03 Proceedings of the 2003 ACM SIGMETRICS international conference on Measurement and modeling of computer systems
Stable distributions, pseudorandom generators, embeddings and data stream computation

FOCS '00 Proceedings of the 41st Annual Symposium on Foundations of Computer Science
Automatically inferring patterns of resource consumption in network traffic

Proceedings of the 2003 conference on Applications, technologies, architectures, and protocols for computer communications
Adaptive filters for continuous queries over distributed data streams

Proceedings of the 2003 ACM SIGMOD international conference on Management of data
Sketch-based change detection: methods, evaluation, and applications

Proceedings of the 3rd ACM SIGCOMM conference on Internet measurement
Holistic UDAFs at streaming speeds

SIGMOD '04 Proceedings of the 2004 ACM SIGMOD international conference on Management of data
On scalable attack detection in the network

Proceedings of the 4th ACM SIGCOMM conference on Internet measurement
Change-Point Monitoring for the Detection of DoS Attacks

IEEE Transactions on Dependable and Secure Computing
A DoS Resilient Flow-level Intrusion Detection Approach for High-speed Networks

ICDCS '06 Proceedings of the 26th IEEE International Conference on Distributed Computing Systems
Very fast containment of scanning worms

SSYM'04 Proceedings of the 13th conference on USENIX Security Symposium - Volume 13
Approximate frequency counts over data streams

VLDB '02 Proceedings of the 28th international conference on Very Large Data Bases
Finding hierarchical heavy hitters in data streams

VLDB '03 Proceedings of the 29th international conference on Very large data bases - Volume 29

Finding frequent items in data streams

Proceedings of the VLDB Endowment
Methods for finding frequent items in data streams

The VLDB Journal — The International Journal on Very Large Data Bases
Fast Manhattan sketches in data streams

Proceedings of the twenty-ninth ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems
A new data streaming method for locating hosts with large connection degree

GLOBECOM'09 Proceedings of the 28th IEEE conference on Global telecommunications
A scalable, efficient and informative approach for anomaly-based intrusion detection systems: theory and practice

International Journal of Network Management
Sequential hashing: A flexible approach for unveiling significant patterns in high speed networks

Computer Networks: The International Journal of Computer and Telecommunications Networking
Dynamic feature analysis and measurement for large-scale network traffic monitoring

IEEE Transactions on Information Forensics and Security
Monitoring abnormal network traffic based on blind source separation approach

Journal of Network and Computer Applications
What's the difference?: efficient set reconciliation without prior context

Proceedings of the ACM SIGCOMM 2011 conference
Accelerating Sketch-Based Computations with GPU: A Case Study for Network Traffic Change Detection

Proceedings of the 2011 ACM/IEEE Seventh Symposium on Architectures for Networking and Communications Systems
Virtual indexing based methods for estimating node connection degrees

Computer Networks: The International Journal of Computer and Telecommunications Networking
Synopses for Massive Data: Samples, Histograms, Wavelets, Sketches

Foundations and Trends in Databases
Flooding attacks detection in backbone traffic using power divergence

Proceedings of the 7th ACM workshop on Performance monitoring and measurement of heterogeneous wireless and wired networks

Quantified Score

Hi-index	0.00

Visualization

Abstract

A key function for network traffic monitoring and analysis is the ability to perform aggregate queries over multiple data streams. Change detection is an important primitive which can be extended to construct many aggregate queries. The recently proposed sketches are among the very few that can detect heavy changes online for high speed links, and thus support various aggregate queries in both temporal and spatial domains. However, it does not preserve the keys (e. g., source IP address) of flows, making it difficult to reconstruct the desired set of anomalous keys. To address this challenge, we propose the reversible sketch data structure along with reverse hashing algorithms to infer the keys of culprit flows. There are two phases. The first operates online, recording the packet stream in a compact representation with negligible extra memory and few extra memory accesses. Our prototype single FPGA board implementation can achieve a throughput of over 16 Gb/s for 40-byte packet streams (the worst case). The second phase identifies heavy changes and their keys from the representation in nearly real time. We evaluate our scheme using traces from large edge routers with OC-12 or higher links. Both the analytical and experimental results show that we are able to achieve online traffic monitoring and accurate change/intrusion detection over massive data streams on high speed links, all in a manner that scales to large key space size. To the best of our knowledge, our system is the first to achieve these properties simultaneously.