Latency modeling and minimization for large-scale scientific workflows in distributed network environments

Authors:
Qishi Wu;Yi Gu;Yuchen Liao;Xukang Lu;Yunyue Lin;Nageswara S. V. Rao
Affiliations:
The University of Memphis, Memphis, TN;The University of Memphis, Memphis, TN;The University of Memphis, Memphis, TN;The University of Memphis, Memphis, TN;The University of Memphis, Memphis, TN;Oak Ridge National Laboratory, Oak Ridge, TN
Venue:
Proceedings of the 44th Annual Simulation Symposium
Year:
2011

Citing 20
Cited 3

Analysis and evaluation of heuristic methods for static task scheduling

Journal of Parallel and Distributed Computing
Dynamic Critical-Path Scheduling: An Effective Technique for Allocating Task Graphs to Multiprocessors

IEEE Transactions on Parallel and Distributed Systems
Task matching and scheduling in heterogeneous computing environments using a genetic-algorithm-based approach

Journal of Parallel and Distributed Computing - Special issue on parallel evolutionary computing
Static scheduling algorithms for allocating directed task graphs to multiprocessors

ACM Computing Surveys (CSUR)
A comparison of eleven static heuristics for mapping a class of independent tasks onto heterogeneous distributed computing systems

Journal of Parallel and Distributed Computing
Performance-Effective and Low-Complexity Task Scheduling for Heterogeneous Computing

IEEE Transactions on Parallel and Distributed Systems
A Compile-Time Scheduling Heuristic for Interconnection-Constrained Heterogeneous Processor Architectures

IEEE Transactions on Parallel and Distributed Systems
A Generalized Scheme for Mapping Parallel Algorithms

IEEE Transactions on Parallel and Distributed Systems
A Comparison of Heuristics for Scheduling DAGs on Multiprocessors

Proceedings of the 8th International Symposium on Parallel Processing
Linear time algorithm for isomorphism of planar graphs (Preliminary Report)

STOC '74 Proceedings of the sixth annual ACM symposium on Theory of computing
Resource allocation in a middleware for streaming data

MGC '04 Proceedings of the 2nd workshop on Middleware for grid computing
A Cluster-based Strategy for Scheduling Task on Heterogeneous Processors

SBAC-PAD '04 Proceedings of the 16th Symposium on Computer Architecture and High Performance Computing
Scheduling of scientific workflows in the ASKALON grid environment

ACM SIGMOD Record
Critical-Path and Priority based Algorithms for Scheduling Workflows with Parameter Sweep Tasks on Global Grids

SBAC-PAD '05 Proceedings of the 17th International Symposium on Computer Architecture on High Performance Computing
A Statistical Study of the Performance of a Task Scheduling Algorithm

IEEE Transactions on Computers
A Dynamic Critical Path Algorithm for Scheduling Scientific Workflow Applications on Global Grids

E-SCIENCE '07 Proceedings of the Third IEEE International Conference on e-Science and Grid Computing
Supporting Distributed Application Workflows in Heterogeneous Computing Environments

ICPADS '08 Proceedings of the 2008 14th IEEE International Conference on Parallel and Distributed Systems
Simulation-based analysis of performance dynamics of distributed applications in heterogeneous network environments

SpringSim '09 Proceedings of the 2009 Spring Simulation Multiconference
A task duplication based bottom-up scheduling algorithm for heterogeneous environments

IPDPS'06 Proceedings of the 20th international conference on Parallel and distributed processing
A state-space search approach for optimizing reliability and cost of execution in distributed sensor networks

IWDC'05 Proceedings of the 7th international conference on Distributed Computing

Modeling and simulation of distributed computing workflows in heterogeneous network environments

Simulation
A Distributed Workflow Management System with Case Study of Real-life Scientific Applications on Grids

Journal of Grid Computing
Distributed workflow mapping algorithm for maximized reliability under end-to-end delay constraint

The Journal of Supercomputing

Quantified Score

Hi-index	0.00

Visualization

Abstract

Large-scale e-science applications feature complex workflows consisting of many computing modules. Mapping such workflows in distributed network environments and minimizing their latency are crucial to those applications that require fast system response and prompt user interaction. We model the time cost of each workflow component and design an efficient algorithm to compute the exact end-to-end delay of the entire workflow by explicitly accounting for the resource sharing dynamics. We further propose a workflow mapping approach to minimize the workflow latency using a recursive optimization procedure. The validity of the cost models and the accuracy of the latency computing algorithm are verified in comparison with an approximate solution, a dynamic system simulation program, and a workflow engine deployed in a real network. The performance superiority of the proposed mapping approach is illustrated by extensive simulation-based comparisons with existing algorithms.