Communications of the ACM
Signature-based text retrieval methods: a survey
Data Engineering
The design and implementation of a log-structured file system
ACM Transactions on Computer Systems (TOCS)
Streaming RAID: a disk array management system for video files
MULTIMEDIA '93 Proceedings of the first ACM international conference on Multimedia
Bitmap index design and evaluation
SIGMOD '98 Proceedings of the 1998 ACM SIGMOD international conference on Management of data
Signature files: an access method for documents and its analytical performance evaluation
ACM Transactions on Information Systems (TOIS)
Space/time trade-offs in hash coding with allowable errors
Communications of the ACM
Fast Text Access Methods for Optical and Large Magnetic Disks: Designs and Performance Comparison
VLDB '88 Proceedings of the 14th International Conference on Very Large Data Bases
Design and Implementation of Bitmap Indices for Scientific Data
IDEAS '01 Proceedings of the International Database Engineering & Applications Symposium
Storage Management for Web Proxies
Proceedings of the General Track: 2002 USENIX Annual Technical Conference
A simple algorithm for finding frequent elements in streams and bags
ACM Transactions on Database Systems (TODS)
Dimensions: why do we need a new data handling architecture for sensor networks?
ACM SIGCOMM Computer Communication Review
Gigascope: a stream database for network applications
Proceedings of the 2003 ACM SIGMOD international conference on Management of data
Operational experiences with high-volume network intrusion detection
Proceedings of the 11th ACM conference on Computer and communications security
HyLog: A High Performance Approach to Managing Disk Layout
FAST '04 Proceedings of the 3rd USENIX Conference on File and Storage Technologies
TSAR: a two tier sensor storage architecture using interval skip graphs
Proceedings of the 3rd international conference on Embedded networked sensor systems
The 8 requirements of real-time stream processing
ACM SIGMOD Record
Advanced Indexing Techniques for Wide-Area Network Monitoring
ICDEW '05 Proceedings of the 21st International Conference on Data Engineering Workshops
File system logging versus clustering: a performance comparison
TCON'95 Proceedings of the USENIX 1995 Technical Conference Proceedings
Scalability in the XFS file system
ATEC '96 Proceedings of the 1996 annual conference on USENIX Annual Technical Conference
Enriching network security analysis with time travel
Proceedings of the ACM SIGCOMM 2008 conference on Data communication
Proceedings of the 10th ACM/IFIP/USENIX International Conference on Middleware
Middleware'09 Proceedings of the ACM/IFIP/USENIX 10th international conference on Middleware
Cheap and large CAMs for high performance data-intensive networked systems
NSDI'10 Proceedings of the 7th USENIX conference on Networked systems design and implementation
OFRewind: enabling record and replay troubleshooting for networks
USENIXATC'11 Proceedings of the 2011 USENIX conference on USENIX annual technical conference
pcapIndex: an index for network packet traces with legacy compatibility
ACM SIGCOMM Computer Communication Review
Collection and exploration of large data monitoring sets using bitmap databases
TMA'10 Proceedings of the Second international conference on Traffic Monitoring and Analysis
A sequence-oriented stream warehouse paradigm for network monitoring applications
PAM'12 Proceedings of the 13th international conference on Passive and Active Measurement
Improving Bandwidth Efficiency for Consistent Multistream Storage
ACM Transactions on Storage (TOS)
Hi-index | 0.00 |
Network monitoring systems that support data archiving and after-the-fact (retrospective) queries are useful for a multitude of purposes, such as anomaly detection and network and security forensics. Data archiving for such systems, however, is complicated by (a) data arrival rates, which may be hundreds of thousands of packets per second on a single link, and (b) the need for online indexing of this data to support retrospective queries. At these data rates, both common database index structures and general-purpose file systems perform poorly. This paper describes Hyperion, a system for archiving, indexing, and on-line retrieval of high-volume data streams. We employ a write-optimized stream file system for high-speed storage of simultaneous data streams, and a novel use of signature file indexes in a distributed multi-level index. We implement Hyperion on commodity hardware and conduct a detailed evaluation using synthetic data and real network traces. Our streaming file system, StreamFS, is shown to be fast enough to archive traces at over a million packets per second. The index allows queries over hours of data to complete in as little as 10-20 seconds, and the entire system is able to index and archive over 200,000 packets/sec while processing simultaneous on-line queries.