Detouring and replication for fast and reliable internet-scale stream processing

  • Authors:
  • Christopher McConnell;Fan Ping;Jeong-Hyon Hwang

  • Affiliations:
  • University at Albany - State University of New York;University at Albany - State University of New York;University at Albany - State University of New York

  • Venue:
  • Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing
  • Year:
  • 2010

Quantified Score

Hi-index 0.00

Visualization

Abstract

iFlow is a replication-based system that can achieve both fast and reliable processing of high volume data streams on the Internet scale. iFlow uses a low degree of replication in conjunction with detouring techniques to overcome network congestion and outages. Computation over iFlow can be expressed as a graph of operators. To cope with varying system conditions these operators continually migrate in a manner that improves performance and availability at the same time. In this paper, we first provide an overview of our iFlow system. Next, we detail how our detouring technique works in the face of network failures to provide high availability for time critical applications. The paper also includes a description of our implementation and preliminary evaluation results demonstrating that iFlow outperforms previous solutions with less overhead. Finally, the paper concludes with our plans for enhancing replication and detouring capabilities.