OSI Reference Model for Telecommunications
OSI Reference Model for Telecommunications
A first-principles approach to understanding the internet's router-level topology
Proceedings of the 2004 conference on Applications, technologies, architectures, and protocols for computer communications
Proceedings of the 2004 conference on Applications, technologies, architectures, and protocols for computer communications
VisFlowConnect: netflow visualizations of link relationships for security situational awareness
Proceedings of the 2004 ACM workshop on Visualization and data mining for computer security
LISA '00 Proceedings of the 14th USENIX conference on System administration
Advanced Message Queuing Protocol
IEEE Internet Computing
MapReduce: simplified data processing on large clusters
Communications of the ACM - 50th anniversary issue: 1958 - 2008
To compress or not to compress - compute vs. IO tradeoffs for mapreduce energy efficiency
Proceedings of the first ACM SIGCOMM workshop on Green networking
Processing flows of information: From data stream to complex event processing
ACM Computing Surveys (CSUR)
Hadoop: The Definitive Guide
Toward scalable internet traffic measurement and analysis with Hadoop
ACM SIGCOMM Computer Communication Review
The Definitive Guide to HTML5 WebSocket
The Definitive Guide to HTML5 WebSocket
| Hi-index | 0.00 |
Collections of network traces have long been used in network traffic analysis. Flow analysis can be used in network anomaly discovery, intrusion detection and more generally, discovery of actionable events on the network. The data collected during processing may be also used for prediction and avoidance of traffic congestion, network capacity planning, and the development of software-defined networking rules. As network flow rates increase and new network technologies are introduced on existing hardware platforms, many organizations find themselves either technically or financially unable to generate, collect, and/or analyze network flow data. The continued rapid growth of network trace data, requires new methods of scalable data collection and analysis. We report on our deployment of a system designed and implemented at the University of Kentucky that supports analysis of network traffic across the enterprise. Our system addresses problems of scale in existing systems, by using distributed computing methodologies, and is based on a combination of stream and batch processing techniques. In addition to collection, stream processing using Storm is utilized to enrich the data stream with ephemeral environment data. Enriched stream-data is then used for event detection and near real-time flow analysis by an in-line complex event processor. Batch processing is performed by the Hadoop MapReduce framework, from data stored in HBase BigTable storage. In benchmarks on our 10 node cluster, using actual network data, we were able to stream process over 315k flows/sec. In batch analysis were we able to process over 2.6M flows/sec with a storage compression ratio of 6.7:1.