Towards jungle computing with Ibis/Constellation

  • Authors:

  • Jason Maassen;Niels Drost;Henri E. Bal;Frank J. Seinstra

  • Affiliations:

  • VU University, Amsterdam, Netherlands;VU University, Amsterdam, Netherlands;VU University, Amsterdam, Netherlands;VU University, Amsterdam, Netherlands

  • Venue:
  • Proceedings of the 2011 workshop on Dynamic distributed data-intensive applications, programming abstractions, and systems
  • Year:
  • 2011

Quantified Score

Hi-index 0.00

Visualization

Abstract

The scientific computing landscape is becoming more and more complex. Besides traditional supercomputers and clusters, scientists can also apply grid and cloud infrastructures. Moreover, the current integration of many-core technologies such as GPUs with such infrastructures adds to the complexity. To make matters worse, data distribution, hardware availability, software heterogeneity, and increasing data sizes, commonly force scientists to use multiple computing platforms simultaneously: a true computing jungle. In this paper we introduce Ibis/Constellation, a software platform specifically designed for distributed, heterogeneous and hierarchical computing environments. In Ibis/Constellation we assume that applications consist of several distinct (but somehow related) activities. These activities can be implemented independently using existing, well understood tools (e.g. MPI, CUDA, etc.). Ibis/Constellation is then used to construct the overall application by coupling the distinct activities. Using application defined labels in combination with context-aware work stealing, Ibis/Constellation provides a simple and efficient mechanism for automatically mapping the activities to the appropriate resources, taking data locality and heterogeneity into account. We show that an existing supernova detection application can be ported to Ibis/Constellation with little effort. By making small changes to the application defined labels, this example application can run efficiently in three very different HPC computing environments: a distributed set of clusters, a large 48-core machine, and a GPU cluster.