Detection and identification of network anomalies using sketch subspaces

Authors:
Xin Li;Fang Bian;Mark Crovella;Christophe Diot;Ramesh Govindan;Gianluca Iannaccone;Anukool Lakhina
Affiliations:
University of Southern California, Los Angeles, CA;University of Southern California, Los Angeles, CA;Boston University, Boston, MA;Thomson Paris Research Lab, Cedex, France;University of Southern California, Los Angeles, CA;Intel Research, Cambridge, UK;Boston University, Boston, MA
Venue:
Proceedings of the 6th ACM SIGCOMM conference on Internet measurement
Year:
2006

Citing 12
Cited 36

Fault detection in an Ethernet network using anomaly signature matching

SIGCOMM '93 Conference proceedings on Communications architectures, protocols and applications
Schemes for fault identification in communication networks

IEEE/ACM Transactions on Networking (TON)
New directions in traffic measurement and accounting

Proceedings of the 2002 conference on Applications, technologies, architectures, and protocols for computer communications
A signal analysis of network traffic anomalies

Proceedings of the 2nd ACM SIGCOMM Workshop on Internet measurment
Proactive Network Fault Detection

INFOCOM '97 Proceedings of the INFOCOM '97. Sixteenth Annual Joint Conference of the IEEE Computer and Communications Societies. Driving the Information Revolution
Sketch-based change detection: methods, evaluation, and applications

Proceedings of the 3rd ACM SIGCOMM conference on Internet measurement
Tabulation based 4-universal hashing with applications to second moment estimation

SODA '04 Proceedings of the fifteenth annual ACM-SIAM symposium on Discrete algorithms
Diagnosing network-wide traffic anomalies

Proceedings of the 2004 conference on Applications, technologies, architectures, and protocols for computer communications
Online identification of hierarchical heavy hitters: algorithms, evaluation, and applications

Proceedings of the 4th ACM SIGCOMM conference on Internet measurement
Mining anomalies using traffic feature distributions

Proceedings of the 2005 conference on Applications, technologies, architectures, and protocols for computer communications
Combining filtering and statistical methods for anomaly detection

IMC '05 Proceedings of the 5th ACM SIGCOMM conference on Internet Measurement
Anomaly detection in IP networks

IEEE Transactions on Signal Processing

Sensitivity of PCA for traffic anomaly detection

Proceedings of the 2007 ACM SIGMETRICS international conference on Measurement and modeling of computer systems
Challenging the supremacy of traffic matrices in anomaly detection

Proceedings of the 7th ACM SIGCOMM conference on Internet measurement
Probabilistic lossy counting: an efficient algorithm for finding heavy hitters

ACM SIGCOMM Computer Communication Review
WebClass: adding rigor to manual labeling of traffic anomalies

ACM SIGCOMM Computer Communication Review
Minimizing collateral damage by proactive surge protection

Proceedings of the 2007 workshop on Large scale attack defense
Extracting hidden anomalies using sketch and non Gaussian multiresolution statistical detection procedures

Proceedings of the 2007 workshop on Large scale attack defense
The eternal sunshine of the sketch data structure

Computer Networks: The International Journal of Computer and Telecommunications Networking
Proactive surge protection: a defense mechanism for bandwidth-based attacks

SS'08 Proceedings of the 17th conference on Security symposium
An image processing approach to traffic anomaly detection

Proceedings of the 4th Asian Conference on Internet Engineering
Detecting distributed network traffic anomaly with network-wide correlation analysis

EURASIP Journal on Advances in Signal Processing - Special issue on signal processing applications in network intrusion detection systems
Beyond Shannon: Characterizing Internet Traffic with Generalized Entropy Metrics

PAM '09 Proceedings of the 10th International Conference on Passive and Active Network Measurement
A distributed data streaming algorithm for network-wide traffic anomaly detection

ACM SIGMETRICS Performance Evaluation Review
ANTIDOTE: understanding and defending against poisoning of anomaly detectors

Proceedings of the 9th ACM SIGCOMM conference on Internet measurement conference
Anomaly extraction in backbone networks using association rules

Proceedings of the 9th ACM SIGCOMM conference on Internet measurement conference
Towards systematic traffic annotation

Proceedings of the 5th international student workshop on Emerging networking experiments and technologies
Flooding attacks detection and victim identification over high speed networks

GIIS'09 Proceedings of the Second international conference on Global Information Infrastructure Symposium
Proactive surge protection: a defense mechanism for bandwidth-based attacks

IEEE/ACM Transactions on Networking (TON)
Online anomaly detection using KDE

GLOBECOM'09 Proceedings of the 28th IEEE conference on Global telecommunications
Optimal volume anomaly detection and isolation in large-scale IP networks using coarse-grained measurements

Computer Networks: The International Journal of Computer and Telecommunications Networking
URCA: pulling out anomalies by their root causes

INFOCOM'10 Proceedings of the 29th conference on Information communications
Measurouting: a framework for routing assisted traffic monitoring

INFOCOM'10 Proceedings of the 29th conference on Information communications
ASTUTE: detecting a different class of traffic anomalies

Proceedings of the ACM SIGCOMM 2010 conference
A scalable, efficient and informative approach for anomaly-based intrusion detection systems: theory and practice

International Journal of Network Management
Temporally oblivious anomaly detection on large networks using functional peers

IMC '10 Proceedings of the 10th ACM SIGCOMM conference on Internet measurement
MAWILab: combining diverse anomaly detectors for automated anomaly labeling and performance benchmarking

Proceedings of the 6th International COnference
SEPIA: privacy-preserving aggregation of multi-domain network events and statistics

USENIX Security'10 Proceedings of the 19th USENIX conference on Security
Accurate network anomaly classification with generalized entropy metrics

Computer Networks: The International Journal of Computer and Telecommunications Networking
Privacy-preserving distributed network troubleshooting—bridging the gap between theory and practice

ACM Transactions on Information and System Security (TISSEC)
Uncovering relations between traffic classifiers and anomaly detectors via graph theory

TMA'10 Proceedings of the Second international conference on Traffic Monitoring and Analysis
0day anomaly detection made possible thanks to machine learning

WWIC'10 Proceedings of the 8th international conference on Wired/Wireless Internet Communications
MeasuRouting: a framework for routing assisted traffic monitoring

IEEE/ACM Transactions on Networking (TON)
Applying random projection to the classification of malicious applications using data mining algorithms

Proceedings of the 50th Annual Southeast Regional Conference
Improving an SVD-based combination strategy of anomaly detectors for traffic labelling

Proceedings of the Asian Internet Engineeering Conference
Anomaly extraction in backbone networks using association rules

IEEE/ACM Transactions on Networking (TON)
Per-flow traffic measurement through randomized counter sharing

IEEE/ACM Transactions on Networking (TON)
ADMIRE: Anomaly detection method using entropy-based PCA with three-step sketches

Computer Communications

Quantified Score

Hi-index	0.00

Visualization

Abstract

Network anomaly detection using dimensionality reduction techniques has received much recent attention in the literature. For example, previous work has aggregated netflow records into origin-destination (OD) flows, yielding a much smaller set of dimensions which can then be mined to uncover anomalies. However, this approach can only identify which OD flow is anomalous, not the particular IP flow(s) responsible for the anomaly. In this paper we show how one can use random aggregations of IP flows (i.e., sketches) to enable more precise identification of the underlying causes of anomalies. We show how to combine traffic sketches with a subspace method to (1) detect anomalies with high accuracy and (2) identify the IP flows(s) that are responsible for the anomaly. Our method has detection rates comparable to previous methods and detects many more anomalies than prior work, taking us a step closer towards a robust on-line system for anomaly detection and identification.