Cost optimized provisioning of elastic resources for application workflows

  • Authors:

  • Eun-Kyu Byun;Yang-Suk Kee;Jin-Soo Kim;Seungryoul Maeng

  • Affiliations:

  • Department of Computer Science, Korea Advanced Institute of Science and Technology, Daejeon 305-701, South Korea;Oracle USA Inc., Redwood Shores, CA 94065, USA;School of Information and Communication Eng., Sungkyunkwan University, Suwon, Gyeonggi-do 440-746, South Korea;Department of Computer Science, Korea Advanced Institute of Science and Technology, Daejeon 305-701, South Korea

  • Venue:
  • Future Generation Computer Systems
  • Year:
  • 2011

Quantified Score

Hi-index 0.00

Visualization

Abstract

Workflow technologies have become a major vehicle for easy and efficient development of scientific applications. In the meantime, state-of-the-art resource provisioning technologies such as cloud computing enable users to acquire computing resources dynamically and elastically. A critical challenge in integrating workflow technologies with resource provisioning technologies is to determine the right amount of resources required for the execution of workflows in order to minimize the financial cost from the perspective of users and to maximize the resource utilization from the perspective of resource providers. This paper suggests an architecture for the automatic execution of large-scale workflow-based applications on dynamically and elastically provisioned computing resources. Especially, we focus on its core algorithm named PBTS (Partitioned Balanced Time Scheduling), which estimates the minimum number of computing hosts required to execute a workflow within a user-specified finish time. The PBTS algorithm is designed to fit both elastic resource provisioning models such as Amazon EC2 and malleable parallel application models such as MapReduce. The experimental results with a number of synthetic workflows and several real science workflows demonstrate that PBTS estimates the resource capacity close to the theoretical low bound.