Using cross-layer adaptations for dynamic data management in large scale coupled scientific workflows

  • Authors:
  • Tong Jin;Fan Zhang;Qian Sun;Hoang Bui;Manish Parashar;Hongfeng Yu;Scott Klasky;Norbert Podhorszki;Hasan Abbasi

  • Affiliations:
  • Rutgers University, Piscataway, NJ;Rutgers University, Piscataway, NJ;Rutgers University, Piscataway, NJ;Rutgers University, Piscataway, NJ;Rutgers University, Piscataway, NJ;University of Nebraska-Lincoln, Lincoln, NE;Oak Ridge National Labortory, Oak Ridge, TN;Oak Ridge National Labortory, Oak Ridge, TN;Oak Ridge National Labortory, Oak Ridge, TN

  • Venue:
  • SC '13 Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis
  • Year:
  • 2013

Quantified Score

Hi-index 0.00

Visualization

Abstract

As system scales and application complexity grow, managing and processing simulation data has become a significant challenge. While recent approaches based on data staging and in-situ/in-transit data processing are promising, dynamic data volumes and distributions, such as those occurring in AMR-based simulations, make the efficient use of these techniques challenging. In this paper we propose cross-layer adaptations that address these challenges and respond at runtime to dynamic data management requirements. Specifically we explore (1) adaptations of the spatial resolution at which the data is processed, (2) dynamic placement and scheduling of data processing kernels, and (3) dynamic allocation of in-transit resources. We also exploit coordinated approaches that dynamically combine these adaptations at the different layers. We evaluate the performance of our adaptive cross-layer management approach on the Intrepid IBM-BlueGene/P and Titan Cray-XK7 systems using Chombo-based AMR applications, and demonstrate its effectiveness in improving overall time-to-solution and increasing resource efficiency.