Challenges in executing large parameter sweep studies across widely distributed computing environments

Authors:
Edward Walker;Chona Guiang
Affiliations:
The University of Texas at Austin, Austin, TX;The University of Texas at Austin, Austin, TX
Venue:
Proceedings of the 5th IEEE workshop on Challenges of large applications in distributed environments
Year:
2007

Citing 3
Cited 10

Matchmaking: Distributed Resource Management for High Throughput Computing

HPDC '98 Proceedings of the 7th IEEE International Symposium on High Performance Distributed Computing
A distributed file system for a wide-area high performance computing infrastructure

WORLDS'06 Proceedings of the 3rd conference on USENIX Workshop on Real, Large Distributed Systems - Volume 3
Personal adaptive clusters as containers for scientific jobs

Cluster Computing

Creating private network overlays for high performance scientific computing

Proceedings of the ACM/IFIP/USENIX 2007 International Conference on Middleware
Exploring many task computing in scientific workflows

Proceedings of the 2nd Workshop on Many-Task Computing on Grids and Supercomputers
Creating private network overlays for high performance scientific computing

MIDDLEWARE2007 Proceedings of the 8th ACM/IFIP/USENIX international conference on Middleware
Grids and Clouds: Making Workflow Applications Work in Heterogeneous Distributed Environments

International Journal of High Performance Computing Applications
Data parallelism in bioinformatics workflows using Hydra

Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing
An adaptive parallel execution strategy for cloud-based scientific workflows

Concurrency and Computation: Practice & Experience
A Provenance-based Adaptive Scheduling Heuristic for Parallel Scientific Workflows in Clouds

Journal of Grid Computing
Evaluating parameter sweep workflows in high performance computing

Proceedings of the 1st ACM SIGMOD Workshop on Scalable Workflow Execution Engines and Technologies
Performance evaluation of parallel strategies in public clouds: A study with phylogenomic workflows

Future Generation Computer Systems
Designing a parallel cloud based comparative genomics workflow to improve phylogenetic analyses

Future Generation Computer Systems

Quantified Score

Hi-index	0.00

Visualization

Abstract

Large parameter sweep studies are common in a broad range of scientific disciplines. However, many challenges exist in supporting this type of computation in a widely distributed computing environment. These challenges exist because contributing sites in a federated distributed computing environment usually expose only a very narrow resource-sharing interface. This paper looks at the challenges encountered by parameter sweep studies using two concrete application examples. The paper also shows how a system for building personal clusters on demand has been developed to solve many of these problems.