Towards flexible exascale stream processing system simulation

  • Authors:
  • Alfred J Park;Cheng-Hong Li;Ravi Nair;Nobuyuki Ohba;Uzi Shvadron;Ayal Zaks;Eugen Schenfeld

  • Affiliations:
  • IBM Research, T.J. Watson Research Center, Yorktown Heights, NY, USA;IBM Research, T.J. Watson Research Center, Yorktown Heights, NY, USA;IBM Research, T.J. Watson Research Center, Yorktown Heights, NY, USA;IBM Research, Tokyo Research Laboratory, Yamato City, Kanagawa, Japan;IBM Research, Haifa Research Laboratory, Haifa University, Mount Carmel, Israel;IBM Research, Haifa Research Laboratory, Haifa University, Mount Carmel, Israel;IBM Research, T.J. Watson Research Center, Yorktown Heights, NY, USA

  • Venue:
  • Simulation
  • Year:
  • 2012

Quantified Score

Hi-index 0.00

Visualization

Abstract

Stream processing is an important emerging computational model for performing complex operations on and across multi-source, high-volume, unpredictable dataflows. We present Flow, a platform for parallel and distributed stream processing system simulation that provides a flexible modeling environment for analyzing stream processing applications. The Flow stream processing system simulator is a high-performance, scalable simulator that automatically parallelizes chunks of the model space and incurs near-zero synchronization overhead for acyclic stream application graphs. We show promising parallel and distributed event rates exceeding 149 million events per second on a cluster with 512 processor cores.