Modelling and programming stream-based distributed computing based on the meta-pipeline approach

  • Authors:
  • Shinichi Yamagiwa;Leonel Sousa

  • Affiliations:
  • INESC-ID/IST, Lisboa, Portugal;INESC-ID/IST, Lisboa, Portugal

  • Venue:
  • International Journal of Parallel, Emergent and Distributed Systems - Advances in Parallel and Distributed Computational Models
  • Year:
  • 2009

Quantified Score

Hi-index 0.00

Visualization

Abstract

Caravela is a computing platform that has been designed in academia to implement distributed stream-based computing based on the flow-model. This paper is focused on the new meta-pipeline execution mechanism, which allows to set up a pipeline of flow-models to be executed in parallel but with data streams flowing in sequence along them. A model and programming tools are presented to develop the proposed meta-pipeline in the Caravela platform. These tools are important, namely to tackle the deadlock problem that can occur due to uninitialised input data streams. A new efficient algorithm is also proposed to identify off-line the uninitialised edges in a 'pipeline graph'. Moreover, optimisation techniques are also proposed to reduce the computation time and the amount of memory required to implement the proposed algorithm, and a graphical user interface (GUI) was implemented to facilitate the programming of applications with the meta-pipeline approach. The experimental results presented in this paper show the practical interest of the meta-pipeline approach for stream-based computing and the effectiveness of both the proposed algorithm and the developed GUI for programming the Caravela platform.