Enabling Real-Time Querying of Live and Historical Stream Data

Authors:
Frederick Reiss;Kurt Stockinger;Kesheng Wu;Arie Shoshani;Joseph M. Hellerstein
Affiliations:
Lawrence Berkeley National Laboratory, USA/ University of California, Berkeley, USA/ IBM Almaden Research Center, USA;Lawrence Berkeley National Laboratory, USA;Lawrence Berkeley National Laboratory, USA;Lawrence Berkeley National Laboratory, USA;University of California, Berkeley, USA
Venue:
SSDBM '07 Proceedings of the 19th International Conference on Scientific and Statistical Database Management
Year:
2007

Citing 0
Cited 17

Replay-based approaches to revision processing in stream query engines

SSPS '08 Proceedings of the 2nd international workshop on Scalable stream processing system
Enriching network security analysis with time travel

Proceedings of the ACM SIGCOMM 2008 conference on Data communication
Real-time GPS via Jamdroid server enhanced by TelegraphCQ & augmented by RFID tag

Proceedings of the International Conference on Advances in Computing, Communication and Control
Information processing using data stream management system on Jamdroid

Proceedings of the International Conference on Advances in Computing, Communication and Control
Improving real-time GPS by incorporating telegraphCQ in Jamdroid architecture

ISWPC'09 Proceedings of the 4th international conference on Wireless pervasive computing
NET-FLi: on-the-fly compression, archiving and indexing of streaming network traffic

Proceedings of the VLDB Endowment
OFRewind: enabling record and replay troubleshooting for networks

USENIXATC'11 Proceedings of the 2011 USENIX conference on USENIX annual technical conference
Efficiently correlating complex events over live and archived data streams

Proceedings of the 5th ACM international conference on Distributed event-based system
A metadata encoding for memory-constrained devices

Proceedings of the 49th Annual Southeast Regional Conference
Event detection over live and archived streams

WAIM'11 Proceedings of the 12th international conference on Web-age information management
Collection and exploration of large data monitoring sets using bitmap databases

TMA'10 Proceedings of the Second international conference on Traffic Monitoring and Analysis
Execution mining

VEE '12 Proceedings of the 8th ACM SIGPLAN/SIGOPS conference on Virtual Execution Environments
Living in the present: on-the-fly information processing in scalable web architectures

Proceedings of the 2nd International Workshop on Cloud Computing Platforms
A sequence-oriented stream warehouse paradigm for network monitoring applications

PAM'12 Proceedings of the 13th international conference on Passive and Active Measurement
Real-time creation of bitmap indexes on streaming network data

The VLDB Journal — The International Journal on Very Large Data Bases
Toward efficient querying of compressed network payloads

USENIX ATC'12 Proceedings of the 2012 USENIX conference on Annual Technical Conference
Towards benchmarking stream data warehouses

Proceedings of the fifteenth international workshop on Data warehousing and OLAP

Quantified Score

Hi-index	0.00

Visualization

Abstract

Applications that query data streams in order to identify trends, patterns, or anomalies can often benefit from comparing the live stream data with archived historical stream data. However, searching this historical data in real time has been considered so far to be prohibitively expensive. One of the main bottlenecks is the update costs of the indices over the archived data. In this paper, we address this problem by using our highly-efficient bitmap indexing technology (called FastBit) and demonstrate that the index update operations are sufficiently efficient for this bottleneck to be removed. We describe our prototype system based on the TelegraphCQ streaming query processor and the FastBit bitmap index. We present a detailed performance evaluation of our system using a complex query workload for analyzing real network traffic data. The combined system uses TelegraphCQ to analyze streams of traffic information and FastBit to correlate current behaviors with historical trends. We demonstrate that our system can simultaneously analyze (1) live streams with high data rates and (2) a large repository of historical stream data.