Flow: A Stream Processing System Simulator

  • Authors:
  • Alfred J. Park; Cheng-Hong Li;Ravi Nair;Nobuyuki Ohba;Uzi Shvadron;Ayal Zaks;Eugen Schenfeld

  • Affiliations:
  • T.J. Watson Res. Center, IBM Res., Yorktown Heights, NY, USA;T.J. Watson Res. Center, IBM Res., Yorktown Heights, NY, USA;T.J. Watson Res. Center, IBM Res., Yorktown Heights, NY, USA;IBM Res., Tokyo Res. Lab., Yamato, Japan;Haifa Res. Lab., Haifa Univ., Haifa, Israel;Haifa Res. Lab., Haifa Univ., Haifa, Israel;T.J. Watson Res. Center, IBM Res., Yorktown Heights, NY, USA

  • Venue:
  • PADS '10 Proceedings of the 2010 IEEE Workshop on Principles of Advanced and Distributed Simulation
  • Year:
  • 2010

Quantified Score

Hi-index 0.00

Visualization

Abstract

Stream processing is an important emerging computational model for performing complex operations on and across multi-source, high volume, unpredictable dataflows. We present Flow, a platform for parallel and distributed stream processing system simulation that provides a flexible modeling environment for analyzing stream processing applications. The Flow stream processing system simulator is a high performance, scalable simulator that automatically parallelizes chunks of the model space and incurs near zero synchronization overhead for stream application graphs that exhibit feed-forward behavior. We show promising multi-threaded and multi-process event rates exceeding 80 million events per second on a cluster with 256 processor cores.