On an integrated mapping and scheduling solution to large-scale scientific workflows in resource sharing environments

  • Authors:

  • Daqing Yun;Qishi Wu;Yi Gu;Xiyang Liu

  • Affiliations:

  • University of Memphis, Memphis, TN;University of Memphis, Memphis, TN;University of Tennessee, Martin, TN;XiDian University, Xi'an, Shaanxi, China

  • Venue:
  • Proceedings of the 46th Annual Simulation Symposium
  • Year:
  • 2013

Quantified Score

Hi-index 0.00

Visualization

Abstract

Next-generation e-science applications feature large-scale data-intensive workflows comprised of many interrelated computing modules. The end-to-end performance of such scientific workflows depends on both the mapping scheme, which determines module assignment, and the scheduling policy, which determines resource allocation if multiple modules are mapped to the same node. These two aspects of workflow optimization are traditionally treated as two separated topics, and the interactions between them have not been fully explored by any existing efforts. As the scale of scientific workflows and the complexity of network environments rapidly increase, each individual aspect of performance optimization alone can only meet with limited success. We conduct an in-depth investigation into workflow execution dynamics of both mapping and scheduling, and propose an integrated solution, referred to as Mapping and Scheduling Interaction (MSI), to achieve a higher level of resource utilization and workflow performance. The efficacy of MSI is illustrated by extensive simulation-based workflow experiments.