Cost-effective cloud HPC resource provisioning by building semi-elastic virtual clusters

  • Authors:
  • Shuangcheng Niu;Jidong Zhai;Xiaosong Ma;Xiongchao Tang;Wenguang Chen

  • Affiliations:
  • Tsinghua University, Beijing, China and Tsinghua University in Shenzhen, Shenzhen, China;Tsinghua University, Beijing, China;North Carolina State University and Oak Ridge National Laboratory;Tsinghua University, Beijing, China;Tsinghua University, Beijing, China and Tsinghua University in Shenzhen, Shenzhen, China

  • Venue:
  • SC '13 Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis
  • Year:
  • 2013

Quantified Score

Hi-index 0.00

Visualization

Abstract

Recent studies have found cloud environments increasingly appealing for executing HPC applications, including tightly coupled parallel simulations. While public clouds offer elastic, on-demand resource provisioning and pay-as-you-go pricing, individual users setting up their on-demand virtual clusters may not be able to take full advantage of common cost-saving opportunities, such as reserved instances. In this paper, we propose a Semi-Elastic Cluster (SEC) computing model for organizations to reserve and dynamically resize a virtual cloud-based cluster. We present a set of integrated batch scheduling plus resource scaling strategies uniquely enabled by SEC, as well as an online reserved instance provisioning algorithm based on job history. Our trace-driven simulation results show that such a model has a 61.0% cost saving than individual users acquiring and managing cloud resources without causing longer average job wait time. Meanwhile, the overhead of acquiring/maintaining shared cloud instances is shown to take only a few seconds.