A linear-time probabilistic counting algorithm for database applications
ACM Transactions on Database Systems (TODS)
Numerical recipes in C (2nd ed.): the art of scientific computing
Numerical recipes in C (2nd ed.): the art of scientific computing
Space/time trade-offs in hash coding with allowable errors
Communications of the ACM
SEDA: an architecture for well-conditioned, scalable internet services
SOSP '01 Proceedings of the eighteenth ACM symposium on Operating systems principles
Counting Distinct Elements in a Data Stream
RANDOM '02 Proceedings of the 6th International Workshop on Randomization and Approximation Techniques
TelegraphCQ: continuous dataflow processing
Proceedings of the 2003 ACM SIGMOD international conference on Management of data
Bitmap algorithms for counting active flows on high speed links
Proceedings of the 3rd ACM SIGCOMM conference on Internet measurement
Flow sampling under hard resource constraints
Proceedings of the joint international conference on Measurement and modeling of computer systems
Proceedings of the 2004 conference on Applications, technologies, architectures, and protocols for computer communications
A robust system for accurate real-time summaries of internet traffic
SIGMETRICS '05 Proceedings of the 2005 ACM SIGMETRICS international conference on Measurement and modeling of computer systems
Ranking flows from sampled traffic
CoNEXT '05 Proceedings of the 2005 ACM conference on Emerging network experiment and technology
Declarative Network Monitoring with an Underprovisioned Query Processor
ICDE '06 Proceedings of the 22nd International Conference on Data Engineering
Monitoring streams: a new class of data management applications
VLDB '02 Proceedings of the 28th international conference on Very Large Data Bases
Load shedding in a data stream manager
VLDB '03 Proceedings of the 29th international conference on Very large data bases - Volume 29
Robust network monitoring in the presence of non-cooperative traffic queries
Computer Networks: The International Journal of Computer and Telecommunications Networking
Distributed scheduling in large scale monitoring infrastructures
CoNEXT '08 Proceedings of the 2008 ACM CoNEXT Conference
Counting Flows over Sliding Windows in High Speed Networks
NETWORKING '09 Proceedings of the 8th International IFIP-TC 6 Networking Conference
Do you know your IQ?: a research agenda for information quality in systems
ACM SIGMETRICS Performance Evaluation Review
Proceedings of the Third European Workshop on System Security
International Journal of Network Management
LISP-TREE: a DNS hierarchy to support the lisp mapping system
IEEE Journal on Selected Areas in Communications - Special issue title on scaling the internet routing system: an interim report
Analysis of the impact of sampling on NetFlow traffic classification
Computer Networks: The International Journal of Computer and Telecommunications Networking
A practical approach to portscan detection in very high-speed links
PAM'11 Proceedings of the 12th international conference on Passive and active measurement
Predictive resource management of multiple monitoring applications
IEEE/ACM Transactions on Networking (TON)
Sketching the delay: tracking temporally uncorrelated flow-level latencies
Proceedings of the 2011 ACM SIGCOMM conference on Internet measurement conference
Validation and improvement of the lossy difference aggregator to measure packet delays
TMA'10 Proceedings of the Second international conference on Traffic Monitoring and Analysis
Measurement Based Analysis of One-Click File Hosting Services
Journal of Network and Systems Management
Tolerating overload attacks against packet capturing systems
USENIX ATC'12 Proceedings of the 2012 USENIX conference on Annual Technical Conference
Scap: stream-oriented network traffic capture and analysis for high-speed networks
Proceedings of the 2013 conference on Internet measurement conference
Mantis: automatic performance prediction for smartphone applications
USENIX ATC'13 Proceedings of the 2013 USENIX conference on Annual Technical Conference
Hi-index | 0.00 |
Monitoring and mining real-time network data streams is crucial for managing and operating data networks. The information that network operators desire to extract from the network traffic is of different size, granularity and accuracy depending on the measurement task (e.g., relevant data for capacity planning and intrusion detection are very different). To satisfy these different demands, a new class of monitoring systems is emerging to handle multiple arbitrary and continuous traffic queries. Such systems must cope with the effects of overload situations due to the large volumes, high data rates and bursty nature of the network traffic. In this paper, we present the design and evaluation of a system that can shed excess load in the presence of extreme traffic conditions, while maintaining the accuracy of the traffic queries within acceptable levels. The main novelty of our approach is that it is able to operate without explicit knowledge of the traffic queries. Instead, it extracts a set of features from the traffic streams to build an on-line predictionmodel of the query resource requirements. This way the monitoring system preserves a high degree of flexibility, increasing the range of applications and network scenarios where it can be used. We implemented our scheme in an existing network monitoring system and deployed it in a research ISP network. Our results show that the system predicts the resources required to run each traffic query with errors below 5%, and that it can efficiently handle extreme load situations, preventing uncontrolled packet losses, with minimum impact on the accuracy of the queries' results.