Provisioning spot market cloud resources to create cost-effective virtual clusters

Authors:
William Voorsluys;Saurabh Kumar Garg;Rajkumar Buyya
Affiliations:
Cloud Computing and Distributed Systems Laboratory, Department of Computer Science and Software Engineering, The University of Melbourne, Australia;Cloud Computing and Distributed Systems Laboratory, Department of Computer Science and Software Engineering, The University of Melbourne, Australia;Cloud Computing and Distributed Systems Laboratory, Department of Computer Science and Software Engineering, The University of Melbourne, Australia
Venue:
ICA3PP'11 Proceedings of the 11th international conference on Algorithms and architectures for parallel processing - Volume Part I
Year:
2011

Citing 7
Cited 5

A Model for Moldable Supercomputer Jobs

IPDPS '01 Proceedings of the 15th International Parallel & Distributed Processing Symposium
A Model For Speedup of Parallel Programs

A Model For Speedup of Parallel Programs
Evaluating the cost-benefit of using cloud computing to extend the capacity of clusters

Proceedings of the 18th ACM international symposium on High performance distributed computing
Reducing Costs of Spot Instances via Checkpointing in the Amazon Elastic Compute Cloud

CLOUD '10 Proceedings of the 2010 IEEE 3rd International Conference on Cloud Computing
Managing Peak Loads by Leasing Cloud Infrastructure Services from a Spot Market

HPCC '10 Proceedings of the 2010 IEEE 12th International Conference on High Performance Computing and Communications
Using inaccurate estimates accurately

JSSPP'10 Proceedings of the 15th international conference on Job scheduling strategies for parallel processing
Modeling user runtime estimates

JSSPP'05 Proceedings of the 11th international conference on Job Scheduling Strategies for Parallel Processing

Performance analysis of HPC applications in the cloud

Future Generation Computer Systems
Characterizing spot price dynamics in public cloud environments

Future Generation Computer Systems
Cost-effective cloud HPC resource provisioning by building semi-elastic virtual clusters

SC '13 Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis
Banking on decoupling: budget-driven sustainability for HPC applications on auction-based clouds

ACM SIGOPS Operating Systems Review
Deconstructing Amazon EC2 Spot Instance Pricing

ACM Transactions on Economics and Computation

Quantified Score

Hi-index	0.00

Visualization

Abstract

Infrastructure-as-a-Service providers are offering their unused resources in the form of variable-priced virtual machines (VMs), known as "spot instances", at prices significantly lower than their standard fixed-priced resources. To lease spot instances, users specify a maximum price they are willing to pay per hour and VMs will run only when the current price is lower than the user's bid. This paper proposes a resource allocation policy that addresses the problem of running deadlineconstrained compute-intensive jobs on a pool of composed solely of spot instances, while exploiting variations in price and performance to run applications in a fast and economical way. Our policy relies on job runtime estimations to decide what are the best types of VMs to run each job and when jobs should run. Several estimation methods are evaluated and compared, using trace-based simulations, which take real price variation traces obtained from Amazon Web Services as input, as well as an application trace from the Parallel Workload Archive. Results demonstrate the effectiveness of running computational jobs on spot instances, at a fraction (up to 60% lower) of the price that would normally cost on fixed priced resources.