SOSP '03 Proceedings of the nineteenth ACM symposium on Operating systems principles
Snort - Lightweight Intrusion Detection for Networks
LISA '99 Proceedings of the 13th USENIX conference on System administration
The OSU Flow-tools Package and CISCO NetFlow Logs
LISA '00 Proceedings of the 14th USENIX conference on System administration
FlowScan: A Network Traffic Flow Reporting and Visualization Tool
LISA '00 Proceedings of the 14th USENIX conference on System administration
MapReduce: simplified data processing on large clusters
OSDI'04 Proceedings of the 6th conference on Symposium on Opearting Systems Design & Implementation - Volume 6
Observing slow crustal movement in residential user traffic
CoNEXT '08 Proceedings of the 2008 ACM CoNEXT Conference
Hive: a warehousing solution over a map-reduce framework
Proceedings of the VLDB Endowment
The NIDS cluster: scalable, stateful network intrusion detection on commodity hardware
RAID'07 Proceedings of the 10th international conference on Recent advances in intrusion detection
High speed network traffic analysis with commodity multi-core systems
IMC '10 Proceedings of the 10th ACM SIGCOMM conference on Internet measurement
A hadoop-based packet trace processing tool
TMA'11 Proceedings of the Third international conference on Traffic monitoring and analysis
Detecting DDoS attacks with Hadoop
Proceedings of The ACM CoNEXT Student Workshop
Live traffic monitoring with tstat: capabilities and experiences
WWIC'10 Proceedings of the 8th international conference on Wired/Wireless Internet Communications
Hadoop: The Definitive Guide
A supervised machine learning approach to classify host roles on line using sFlow
Proceedings of the first edition workshop on High performance and programmable networking
Stream-monitoring with blockmon: convergence of network measurements and data analytics platforms
ACM SIGCOMM Computer Communication Review
A big data based data storage systems for rock burst experiment
International Journal of Wireless and Mobile Computing
Scalable hybrid stream and hadoop network analysis system
Proceedings of the 5th ACM/SPEC international conference on Performance engineering
Scalable and Real-Time Deep Packet Inspection
UCC '13 Proceedings of the 2013 IEEE/ACM 6th International Conference on Utility and Cloud Computing
Hi-index | 0.00 |
Internet traffic measurement and analysis has long been used to characterize network usage and user behaviors, but faces the problem of scalability under the explosive growth of Internet traffic and high-speed access. Scalable Internet traffic measurement and analysis is difficult because a large data set requires matching computing and storage resources. Hadoop, an open-source computing platform of MapReduce and a distributed file system, has become a popular infrastructure for massive data analytics because it facilitates scalable data processing and storage services on a distributed computing system consisting of commodity hardware. In this paper, we present a Hadoop-based traffic monitoring system that performs IP, TCP, HTTP, and NetFlow analysis of multi-terabytes of Internet traffic in a scalable manner. From experiments with a 200-node testbed, we achieved 14 Gbps throughput for 5 TB files with IP and HTTP-layer analysis MapReduce jobs. We also explain the performance issues related with traffic analysis MapReduce jobs.