Deriving traffic demands for operational IP networks: methodology and experience
IEEE/ACM Transactions on Networking (TON)
Charging from sampled network usage
IMW '01 Proceedings of the 1st ACM SIGCOMM Workshop on Internet Measurement
Data Compression: The Complete Reference
Data Compression: The Complete Reference
Locality-sensitive hashing scheme based on p-stable distributions
SCG '04 Proceedings of the twentieth annual symposium on Computational geometry
The OSU Flow-tools Package and CISCO NetFlow Logs
LISA '00 Proceedings of the 14th USENIX conference on System administration
More Netflow Tools for Performance and Security
LISA '04 Proceedings of the 18th USENIX conference on System administration
Mining anomalies using traffic feature distributions
Proceedings of the 2005 conference on Applications, technologies, architectures, and protocols for computer communications
Integrating compression and execution in column-oriented database systems
Proceedings of the 2006 ACM SIGMOD international conference on Management of data
RadixZip: linear time compression of token streams
VLDB '07 Proceedings of the 33rd international conference on Very large data bases
Compressed indexing and local alignment of DNA
Bioinformatics
Compressing term positions in web indexes
Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval
Online memory compression for embedded systems
ACM Transactions on Embedded Computing Systems (TECS)
The network data handling war: MySQL vs. NfDump
EUNICE'10 Proceedings of the 16th EUNICE/IFIP WG 6.6 conference on Networked services and applications: engineering, control and management
NetStore: an efficient storage infrastructure for network forensics and monitoring
RAID'10 Proceedings of the 13th international conference on Recent advances in intrusion detection
pcapIndex: an index for network packet traces with legacy compatibility
ACM SIGCOMM Computer Communication Review
Collection and exploration of large data monitoring sets using bitmap databases
TMA'10 Proceedings of the Second international conference on Traffic Monitoring and Analysis
Apache hadoop performance-tuning methodologies and best practices
ICPE '12 Proceedings of the 3rd ACM/SPEC International Conference on Performance Engineering
A universal algorithm for sequential data compression
IEEE Transactions on Information Theory
Real-time creation of bitmap indexes on streaming network data
The VLDB Journal — The International Journal on Very Large Data Bases
Hi-index | 0.00 |
Network traffic archival solutions are fundamental for a number of emerging applications that require: a) efficient storage of high-speed streams of traffic records and b) support for interactive exploration of massive datasets. Compression is a fundamental building block for any traffic archival solution. However, present solutions are tied to general-purpose compressors, which do not exploit patterns of network traffic data and require to decompress a lot of redundant data for high selectivity queries. In this work we introduce RasterZIP, a novel domain-specific compressor designed for network traffic monitoring data. RasterZIP uses an optimized lossless encoding that exploits patterns of traffic data, like the fact that IP addresses tend to share a common prefix. RasterZIP also introduces a novel decompression scheme that accelerates highly selective queries targeting a small portion of the dataset. With our solution we can achieve high-speed on-the-fly compression of more than half a million traffic records per second. We compare RasterZIP with the fastest Lempel-Ziv-based compressor and show that our solution improves the state-of-the-art both in terms of compression ratios and query response times without introducing penalty in any other performance metric.