Online execution time prediction for computationally intensive applications with periodic progress updates

Authors:
Maria Chtepen;Filip H. Claeys;Bart Dhoedt;Filip Turck;Jan Fostier;Piet Demeester;Peter A. Vanrolleghem
Affiliations:
Department of Information Technology Broadband Communication Networks, IBCN, Ghent University, IBBT, Gent, Belgium 9050;MOSTforWATER N.V., Kortrijk, Belgium 8500;Department of Information Technology Broadband Communication Networks, IBCN, Ghent University, IBBT, Gent, Belgium 9050;Department of Information Technology Broadband Communication Networks, IBCN, Ghent University, IBBT, Gent, Belgium 9050;Department of Information Technology Broadband Communication Networks, IBCN, Ghent University, IBBT, Gent, Belgium 9050;Department of Information Technology Broadband Communication Networks, IBCN, Ghent University, IBBT, Gent, Belgium 9050;modelEAU, Département de génie civil et génie des eaux, Pavillon Adrien-Pouliot, Université Laval, Québec, Canada G1V 0A6
Venue:
The Journal of Supercomputing
Year:
2012

Citing 10
Cited 0

A high-performance, portable implementation of the MPI message passing interface standard

Parallel Computing
Statistical Prediction of Task Execution Times through Analytic Benchmarking for Scheduling in a Heterogeneous Environment

IEEE Transactions on Computers
Stochastic Prediction of Execution Time for Dynamic Bulk Synchronous Computations

The Journal of Supercomputing
Run-Time Statistical Estimation of Task Execution Times for Heterogeneous Distributed Computing

HPDC '96 Proceedings of the 5th IEEE International Symposium on High Performance Distributed Computing
The workload on parallel supercomputers: modeling the characteristics of rigid jobs

Journal of Parallel and Distributed Computing
Job Completion Prediction in Grid Using Distributed Case-based Reasoning

WETICE '05 Proceedings of the 14th IEEE International Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprise
Scheduling strategies for mapping application workflows onto the grid

HPDC '05 Proceedings of the High Performance Distributed Computing, 2005. HPDC-14. Proceedings. 14th IEEE International Symposium
Adaptive Task Checkpointing and Replication: Toward Efficient Fault-Tolerant Grids

IEEE Transactions on Parallel and Distributed Systems
Using Templates to Predict Execution Time of Scientific Workflow Applications in the Grid

CCGRID '09 Proceedings of the 2009 9th IEEE/ACM International Symposium on Cluster Computing and the Grid
ScoPred–scalable user-directed performance prediction using complexity modeling and historical data

JSSPP'05 Proceedings of the 11th international conference on Job Scheduling Strategies for Parallel Processing

Quantified Score

Hi-index	0.00

Visualization

Abstract

The effectiveness of distributed execution of computationally intensive applications (jobs) largely depends on the quality of the applied scheduling approach. However, most of the existing non-trivial scheduling algorithms rely on prior knowledge or on prediction of application parameters, such as execution time, size of input and output, dependencies, etc., to assign applications to the available computational resources. A major issue is that these parameters are hard to determine in advance, especially if the end user does not possess an extensive history of previous application runs.In this work we propose an online method for execution time prediction of applications, for which execution progress can be collected at run-time. Using dynamic progress information, the total job execution time can be predicted using extrapolation. However, the predictions achieved by extrapolation are far from precise and often vary over time as a result of changing application dynamics and varying resource load. Therefore, to compute the actual job execution time we match a number of predefined prediction evolution models against the consecutive extrapolations, by adopting nonlinear curve-fitting. The "best-fit" coefficients allow for more accurate execution time prediction.The predictions made are used to enhance a dynamic scheduling algorithm for workflows introduced in our earlier work. The scheduling algorithm is run with and without curve-fitting, showing a performance improvement of up to 15% in the former case.