Computation and communication in R*: a distributed database manager
ACM Transactions on Computer Systems (TOCS)
Volcano— An Extensible and Parallel Query Evaluation System
IEEE Transactions on Knowledge and Data Engineering
Chained Declustering: A New Availability Strategy for Multiprocessor Database Machines
Proceedings of the Sixth International Conference on Data Engineering
Intrusion Detection and Intrusion Prevention on a Large Network: A Case Study
Proceedings of the Workshop on Intrusion Detection and Network Monitoring
A Visual Approach for Monitoring Logs
LISA '98 Proceedings of the 12th USENIX conference on System administration
Gecko: tracking a very large billing system
ATEC '00 Proceedings of the annual conference on USENIX Annual Technical Conference
Refereed Papers: Real-time Log File Analysis Using the Simple Event Correlator (SEC)
LISA '04 Proceedings of the 18th USENIX conference on System administration
Beehive: O(1)lookup performance for power-law query distributions in peer-to-peer overlays
NSDI'04 Proceedings of the 1st conference on Symposium on Networked Systems Design and Implementation - Volume 1
| Hi-index | 0.00 |
Server systems invariably write detailed activity logs whose value is widespread, whether measuring marketing campaigns, detecting operational trends or catching fraud or intrusion. Unfortunately, production volumes overwhelm the capacity and manageability of traditional data management systems, such as relational databases. Just loading 1,000,000 records is a big deal today, to say nothing of the billions of records often seen in high-end network security, network operations and web applications. Since the magnitude of the problem is scaling with increases in CPU and networking speeds, it doesn't help to wait for faster systems to catch up.This paper discusses the issues involving large-scale log management, and describes a new type of data management platform called a Log Management System, which is specifically designed to cost effectively compress, manage and analyze log records in their original, unsummarized form. To quote Tom Lehrer, "I have a modest example here" - in this case commercial software that can store and process logs in parallel across a cluster of Linux-based PCs using a combination of SQL and perl. The paper concludes with some lessons we learned in building the system.