Load Management and High Availability in the Borealis Distributed Stream Processing Engine

Authors:
Nesime Tatbul;Yanif Ahmad;Uğur Çetintemel;Jeong-Hyon Hwang;Ying Xing;Stan Zdonik
Affiliations:
Department of Computer Science, ETH Zürich, Zürich, Switzerland;Department of Computer Science, Brown University, Providence, USA;Department of Computer Science, Brown University, Providence, USA;Department of Computer Science, Brown University, Providence, USA;Department of Computer Science, Brown University, Providence, USA;Department of Computer Science, Brown University, Providence, USA
Venue:
GeoSensor Networks
Year:
2008

Citing 14
Cited 0

Multi-sensor location tracking

MobiCom '98 Proceedings of the 4th annual ACM/IEEE international conference on Mobile computing and networking
Lots o'Ticks: real time high performance time series queries on billions of trades and quotes

SIGMOD '01 Proceedings of the 2001 ACM SIGMOD international conference on Management of data
Aurora: a new model and architecture for data stream management

The VLDB Journal — The International Journal on Very Large Data Bases
Highly available, fault-tolerant, parallel dataflows

SIGMOD '04 Proceedings of the 2004 ACM SIGMOD international conference on Management of data
High-Availability Algorithms for Distributed Stream Processing

ICDE '05 Proceedings of the 21st International Conference on Data Engineering
Dynamic Load Distribution in the Borealis Stream Processor

ICDE '05 Proceedings of the 21st International Conference on Data Engineering
Distributed operation in the Borealis stream processing engine

Proceedings of the 2005 ACM SIGMOD international conference on Management of data
Network-Aware Operator Placement for Stream-Processing Systems

ICDE '06 Proceedings of the 22nd International Conference on Data Engineering
Dealing with Overload in Distributed Stream Processing Systems

ICDEW '06 Proceedings of the 22nd International Conference on Data Engineering Workshops
Adaptive Control of Extreme-scale Stream Processing Systems

ICDCS '06 Proceedings of the 26th IEEE International Conference on Distributed Computing Systems
Providing resiliency to load variations in distributed stream processing

VLDB '06 Proceedings of the 32nd international conference on Very large data bases
Load shedding in a data stream manager

VLDB '03 Proceedings of the 29th international conference on Very large data bases - Volume 29
An integration framework for sensor networks and data stream management systems

VLDB '04 Proceedings of the Thirtieth international conference on Very large data bases - Volume 30
Fast and Reliable Stream Processing over Wide Area Networks

ICDEW '07 Proceedings of the 2007 IEEE 23rd International Conference on Data Engineering Workshop

Quantified Score

Hi-index	0.00

Visualization

Abstract

Borealis is a distributed stream processing engine that has been developed at Brandeis University, Brown University, and MIT. It extends the first generation of data stream processing systems with advanced capabilities such as distributed operation, scalability with time-varying load, high availability against failures, and dynamic data and query modifications. In this paper, we focus on aspects that are related to load management and high availability in Borealis. We describe our algorithms for balanced and resilient load distribution, scalable distributed load shedding, and cooperative and self-configuring high availability. We also present experimental results from our prototype implementation showing the effectiveness of these algorithms.