A novel approach for distributed application scheduling based on prediction of communication events

Authors:
Evgueni Dodonov;Rodrigo Fernandes de Mello
Affiliations:
University of São Paulo, ICMC, São Carlos, SP, Brazil;University of São Paulo, ICMC, São Carlos, SP, Brazil
Venue:
Future Generation Computer Systems
Year:
2010

Citing 56
Cited 6

Radial basis functions for multivariable interpolation: a review

Algorithms for approximation
Predictability of Process Resource Usage: A Measurement-Based Study on UNIX

IEEE Transactions on Software Engineering
Neural networks and physical systems with emergent collective computational abilities

Neurocomputing: foundations of research
A practical method for calculating largest Lyapunov exponents from small data sets

Physica D
The temporal Kohonen map

Neural Networks
Practical prefetching techniques for multiprocessor file systems

Distributed and Parallel Databases - Selected papers from the first international conference on parallel and distributed information systems
Supervised learning extensions to the CLAM network

Neural Networks
Scheduling with implicit information in distributed systems

SIGMETRICS '98/PERFORMANCE '98 Proceedings of the 1998 ACM SIGMETRICS joint international conference on Measurement and modeling of computer systems
The MOSIX multicomputer operating system for high performance cluster computing

Future Generation Computer Systems - Special issue on HPCN '97
Queueing networks and Markov chains: modeling and performance evaluation with computer science applications

Queueing networks and Markov chains: modeling and performance evaluation with computer science applications
Automatic Segmentation of Acoustic Musical Signals Using Hidden Markov Models

IEEE Transactions on Pattern Analysis and Machine Intelligence
The Hierarchical Hidden Markov Model: Analysis and Applications

Machine Learning
Improving Gang Scheduling through job performance analysis and malleability

ICS '01 Proceedings of the 15th international conference on Supercomputing
Condor: a distributed job scheduler

Beowulf cluster computing with Linux
Load Distributing for Locally Distributed Systems

Computer
Segmentation of ultrasound images by using a hybrid neural network

Pattern Recognition Letters
A Historical Application Profiler for Use by Parallel Schedulers

IPPS '97 Proceedings of the Job Scheduling Strategies for Parallel Processing
Theory and Practice in Parallel Job Scheduling

IPPS '97 Proceedings of the Job Scheduling Strategies for Parallel Processing
Predicting Application Run Times Using Historical Information

IPPS/SPDP '98 Proceedings of the Workshop on Job Scheduling Strategies for Parallel Processing
A self-organising network that grows when required

Neural Networks - New developments in self-organizing maps
Estimation of entropy and mutual information

Neural Computation
A Probabilistic RBF Network for Classification

IJCNN '00 Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks (IJCNN'00)-Volume 4 - Volume 4
An Experimental Study of Load Balancing Performance

An Experimental Study of Load Balancing Performance
Learning precise timing with lstm recurrent networks

The Journal of Machine Learning Research
ProGrid: A Proxy-Based Architecture for Grid Operation and Management

SBAC-PAD '03 Proceedings of the 15th Symposium on Computer Architecture and High Performance Computing
Resource Estimation and Task Scheduling for Multithreaded Reconfigurable Architectures

ICPADS '04 Proceedings of the Parallel and Distributed Systems, Tenth International Conference
GridBox: securing hosts from malicious and greedy applications

MGC '04 Proceedings of the 2nd workshop on Middleware for grid computing
PAT: a postmortem object access pattern analysis and visualization tool

CCGRID '04 Proceedings of the 2004 IEEE International Symposium on Cluster Computing and the Grid
A Routing Load Balancing Policy for Grid Computing Environments

AINA '06 Proceedings of the 20th International Conference on Advanced Information Networking and Applications - Volume 01
A Resource Reservation Algorithm for Power-Aware Scheduling of Periodic and Aperiodic Real-Time Tasks

IEEE Transactions on Computers
MJSA: Markov job scheduler based on availability in desktop grid computing environment

Future Generation Computer Systems
Estimation of Hurst exponent revisited

Computational Statistics & Data Analysis
A self-organizing neural network for detecting novelties

Proceedings of the 2007 ACM symposium on Applied computing
RouteGA: A Grid Load Balancing Algorithm with Genetic Support

AINA '07 Proceedings of the 21st International Conference on Advanced Networking and Applications
ULE: a modern scheduler for FreeBSD

BSDC'03 Proceedings of the BSD Conference 2003 on BSD Conference
A Model for Automatic On-Line Process Behavior Extraction, Classification and Prediction in Heterogeneous Distributed Systems

CCGRID '07 Proceedings of the Seventh IEEE International Symposium on Cluster Computing and the Grid
CATS benchmark time series prediction by Kalman smoother with cross-validated noise density

Neurocomputing
Time series prediction with a weighted bidirectional multi-stream extended Kalman filter

Neurocomputing
Transductive support vector machines for structured variables

Proceedings of the 24th international conference on Machine learning
POODLE-L

Bioinformatics
Predict task running time in grid environments based on CPU load predictions

Future Generation Computer Systems
Parallelization of cellular neural networks on GPU

Pattern Recognition
A tale of four kernels

Proceedings of the 30th international conference on Software engineering
Toward an Efficient Middleware for Multithreaded Applications in Computational Grid

CSE '08 Proceedings of the 2008 11th IEEE International Conference on Computational Science and Engineering
An ant algorithm for balanced job scheduling in grids

Future Generation Computer Systems
Dynamic workload balancing of parallel applications with user-level scheduling on the Grid

Future Generation Computer Systems
Extraction and classification of user behavior

EUC'07 Proceedings of the 2007 international conference on Embedded and ubiquitous computing
A network evaluation for LAN, MAN and WAN grid environments

EUC'05 Proceedings of the 2005 international conference on Embedded and Ubiquitous Computing
Adaptive technique for automatic communication access pattern discovery applied to data prefetching in distributed applications using neural networks and stochastic models

ISPA'06 Proceedings of the 4th international conference on Parallel and Distributed Processing and Applications
A dynamic regularized radial basis function network for nonlinear,nonstationary time series prediction

IEEE Transactions on Signal Processing
Self organization of a massive document collection

IEEE Transactions on Neural Networks
Modified ART 2A growing network capable of generating a fixed number of nodes

IEEE Transactions on Neural Networks
Identification and control of dynamical systems using the self-organizing map

IEEE Transactions on Neural Networks
Continuous-time temporal back-propagation with adaptable time delays

IEEE Transactions on Neural Networks
On the computational power of Elman-style recurrent networks

IEEE Transactions on Neural Networks
A complex network-based approach for job scheduling in grid environments

HPCC'07 Proceedings of the Third international conference on High Performance Computing and Communications

Editorial: Special section: Peer-to-peer grid technologies

Future Generation Computer Systems
Defining and applying prediction performance metrics on a recurrent NARX time series model

Neurocomputing
Combining a recurrent neural network and a PID controller for prognostic purpose: a way to improve the accuracy of predictions

WSEAS Transactions on Systems and Control
SLA-based resource provisioning for heterogeneous workloads in a virtualized cloud datacenter

ICA3PP'11 Proceedings of the 11th international conference on Algorithms and architectures for parallel processing - Volume Part I
Classification of time series generation processes using experimental tools: a survey and proposal of an automatic and systematic approach

International Journal of Computational Science and Engineering
Energy-aware resource allocation heuristics for efficient management of data centers for Cloud computing

Future Generation Computer Systems

Quantified Score

Hi-index	0.00

Visualization

Abstract

The evolution of commodity computing lead to the possibility of efficient usage of interconnected machines to solve computationally-intensive tasks, which were previously solvable only by using expensive supercomputers. This, however, required new methods for process scheduling and distribution, considering the network latency, communication cost, heterogeneous environments and distributed computing constraints. An efficient distribution of processes over such environments requires an adequate scheduling strategy, as the cost of inefficient process allocation is unacceptably high. Therefore, a knowledge and prediction of application behavior is essential to perform effective scheduling. In this paper, we overview the evolution of scheduling approaches, focusing on distributed environments. We also evaluate the current approaches for process behavior extraction and prediction, aiming at selecting an adequate technique for online prediction of application execution. Based on this evaluation, we propose a novel model for application behavior prediction, considering chaotic properties of such behavior and the automatic detection of critical execution points. The proposed model is applied and evaluated for process scheduling in cluster and grid computing environments. The obtained results demonstrate that prediction of the process behavior is essential for efficient scheduling in large-scale and heterogeneous distributed environments, outperforming conventional scheduling policies by a factor of 10, and even more in some cases. Furthermore, the proposed approach proves to be efficient for online predictions due to its low computational cost and good precision.