A Hybrid Approach to High Availability in Stream Processing Systems

Authors:
Zhe Zhang;Yu Gu;Fan Ye;Hao Yang;Minkyong Kim;Hui Lei;Zhen Liu
Affiliations:
National Center for Computational Sciences Oak Ridge National Laboratory Oak Ridge, TN;Department of Computer Science and Engineering University of Minnesota Minneapolis, MN;IBM T.J. Watson Research Center Hawthorne, NY;Nokia Research Center White Plains, NY;IBM T.J. Watson Research Center Hawthorne, NY;IBM T.J. Watson Research Center Hawthorne, NY;Nokia Research China Lab Beijing, China
Venue:
ICDCS '10 Proceedings of the 2010 IEEE 30th International Conference on Distributed Computing Systems
Year:
2010

Citing 0
Cited 3

Task Scheduling Algorithm for Multicore Processor System for Minimizing Recovery Time in Case of Single Node Fault

CCGRID '12 Proceedings of the 2012 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (ccgrid 2012)
Integrating scale out and fault tolerance in stream processing using operator state management

Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data
A solution for optimizing recovery time in cloud computing

Proceedings of the 8th International Conference on Ubiquitous Information Management and Communication

Quantified Score

Hi-index	0.00

Visualization

Abstract

Stream processing is widely used by today’s applications such as financial data analysis and disaster response. In distributed stream processing systems, machine fail-stop events are handled by either active standby or passive standby. However, existing high availability (HA) schemes have not sufficiently addressed the situation when a machine becomes temporarily unavailable due to data rate spikes, intensive analysis or job sharing, which happens frequently but lasts for short time. It is not clear how well active and passive standby fare against such transient unavailability. In this paper, we first critically examine the suitability of active and passive standby against transient unavailability in a real testbed environment. We find that both approaches have advantages and drawbacks, but neither is ideal to provide fast recovery at low overhead as required to handle transient unavailability. Based on the insights gained, we propose a novel hybrid HA method that switches between active and passive standby modes depending on the occurrence of failure events. It presents a desirable tradeoff that is different from existing HA approaches: low overhead during normal conditions and fast recovery upon transient or permanent failure events. We have implemented our hybrid method and compared it with existing HA designs with comprehensive evaluation. The results show that our hybrid method can reduce two-thirds of the recovery time compared to passive standby and 80% message overhead compared to active standby, allowing applications to enjoy uninterrupted processing without paying a high premium.