Engineering resource management middleware for optimizing the performance of clouds processing mapreduce jobs with deadlines

Authors:
Norman Lim;Shikharesh Majumdar;Peter Ashwood-Smith
Affiliations:
Carleton University, Ottawa, ON, Canada;Carleton University, Ottawa, ON, Canada;Huawei Technologies Canada, Kanata, ON, Canada
Venue:
Proceedings of the 5th ACM/SPEC international conference on Performance engineering
Year:
2014

Citing 7
Cited 0

Program Does Not Equal Program: Constraint Programming and Its Relationship to Mathematical Programming

Interfaces
Linear Formulation of Constraint Programming Models and Hybrid Solvers

CP '02 Proceedings of the 6th International Conference on Principles and Practice of Constraint Programming
Time-Indexed Formulations for Machine Scheduling Problems: Column Generation

INFORMS Journal on Computing
MapReduce: simplified data processing on large clusters

OSDI'04 Proceedings of the 6th conference on Symposium on Opearting Systems Design & Implementation - Volume 6
Cloud computing and emerging IT platforms: Vision, hype, and reality for delivering computing as the 5th utility

Future Generation Computer Systems
Scheduling Hadoop Jobs to Meet Deadlines

CLOUDCOM '10 Proceedings of the 2010 IEEE Second International Conference on Cloud Computing Technology and Science
Scheduling Mixed Real-Time and Non-real-Time Applications in MapReduce Environment

ICPADS '11 Proceedings of the 2011 IEEE 17th International Conference on Parallel and Distributed Systems

Quantified Score

Hi-index	0.00

Visualization

Abstract

This paper focuses on devising efficient resource management techniques used by the resource management middleware in clouds that handle MapReduce jobs with end-to-end service level agreements (SLAs) comprising an earliest start time, execution time, and a deadline. This research and development work, performed in collaboration with our industrial partner, presents the formulation of the matchmaking and scheduling problem for MapReduce jobs as an optimization problem using: Mixed Integer Linear Programming (MILP) and Constraint Programming (CP) techniques. In addition to the formulations devised, our experience in implementing the MILP and CP models using various open source as well as commercial software packages is described. Furthermore, a performance evaluation of the different approaches used to implement the formulations is conducted using a variety of different workloads.