Experiences using cloud computing for a scientific workflow application

Authors:
Jens-Sönke Vöckler;Gideon Juve;Ewa Deelman;Mats Rynge;Bruce Berriman
Affiliations:
USC, Marina del Rey, CA, USA;USC, Marina del Rey, CA, USA;USC, Marina del Rey, CA, USA;USC, Marina del Rey, CA, USA;Caltech, Pasadena, CA, USA
Venue:
Proceedings of the 2nd international workshop on Scientific cloud computing
Year:
2011

Citing 17
Cited 9

Dynamic Virtual Clusters in a Grid Site Manager

HPDC '03 Proceedings of the 12th IEEE International Symposium on High Performance Distributed Computing
A Case For Grid Computing On Virtual Machines

ICDCS '03 Proceedings of the 23rd International Conference on Distributed Computing Systems
A case for high performance computing with virtual machines

Proceedings of the 20th annual international conference on Supercomputing
Pegasus: A framework for mapping complex scientific workflows onto distributed systems

Scientific Programming
Virtual workspaces: Achieving quality of service and quality of life in the Grid

Scientific Programming - Dynamic Grids and Worldwide Computing
Workflow task clustering for best effort systems with Pegasus

Proceedings of the 15th ACM Mardi Gras conference: From lightweight mash-ups to lambda grids: Understanding the spectrum of distributed computing requirements, applications, tools, infrastructures, interoperability, and the incremental adoption of key capabilities
Xen-Based HPC: A Parallel I/O Perspective

CCGRID '08 Proceedings of the 2008 Eighth IEEE International Symposium on Cluster Computing and the Grid
The cost of doing science on the cloud: the Montage example

Proceedings of the 2008 ACM/IEEE conference on Supercomputing
Contextualization: Providing One-Click Virtual Clusters

ESCIENCE '08 Proceedings of the 2008 Fourth IEEE International Conference on eScience
On the Use of Cloud Computing for Scientific Workflows

ESCIENCE '08 Proceedings of the 2008 Fourth IEEE International Conference on eScience
Can cloud computing reach the top500?

Proceedings of the combined workshops on UnConventional high performance computing workshop plus memory access workshop
Sky Computing

IEEE Internet Computing
High-Performance Cloud Computing: A View of Scientific Applications

ISPAN '09 Proceedings of the 2009 10th International Symposium on Pervasive Systems, Algorithms, and Networks
Data Sharing Options for Scientific Workflows on Amazon EC2

Proceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis
Performance Analysis of High Performance Computing Applications on the Amazon Web Services Cloud

CLOUDCOM '10 Proceedings of the 2010 IEEE Second International Conference on Cloud Computing Technology and Science
Wrangler: virtual cluster provisioning for the cloud

Proceedings of the 20th international symposium on High performance distributed computing
Paravirtualization for HPC systems

ISPA'06 Proceedings of the 2006 international conference on Frontiers of High Performance Computing and Networking

Efficient provisioning of bursty scientific workloads on the cloud using adaptive elasticity control

Proceedings of the 3rd workshop on Scientific Cloud Computing Date
A cost-effective cloud computing framework for accelerating multimedia communication simulations

Journal of Parallel and Distributed Computing
Cost- and deadline-constrained provisioning for scientific workflow ensembles in IaaS clouds

SC '12 Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis
Designing and Deploying a Scientific Computing Cloud Platform

GRID '12 Proceedings of the 2012 ACM/IEEE 13th International Conference on Grid Computing
Parametric Optimization of Linear and Non-Linear Models via Parallel Computing to Enhance Web-Spatial DSS Interactivity

International Journal of Decision Support System Technology
A survey of migration mechanisms of virtual machines

ACM Computing Surveys (CSUR)
Failure analysis of distributed scientific workflows executing in the cloud

Proceedings of the 8th International Conference on Network and Service Management
SIDR: structure-aware intelligent data routing in Hadoop

SC '13 Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis
Evaluating I/O aware network management for scientific workflows on networked clouds

NDM '13 Proceedings of the Third International Workshop on Network-Aware Data Management

Quantified Score

Hi-index	0.00

Visualization

Abstract

Clouds are rapidly becoming an important platform for scientific applications. In this paper we describe our experiences running a scientific workflow application in the cloud. The application was developed to process astronomy data released by the Kepler project, a NASA mission to search for Earth-like planets orbiting other stars. This workflow was deployed across multiple clouds using the Pegasus Workflow Management System. The clouds used include several sites within the FutureGrid, NERSC's Magellan cloud, and Amazon EC2. We describe how the application was deployed, evaluate its performance executing in different clouds (based on Nimbus, Eucalyptus, and EC2), and discuss the challenges of deploying and executing workflows in a cloud environment. We also demonstrate how Pegasus was able to support sky computing by executing a single workflow across multiple cloud infrastructures simultaneously.