Scheduling data processing flows under budget constraint on the cloud

Authors:
Fei Cao;Dabin Ding;Dunren Che;Michelle M. Zhu;Wen-Chi Hou
Affiliations:
Southern Illinois University Carbondale, IL;Southern Illinois University Carbondale, IL;Illinois University Carbondale, IL;Illinois University Carbondale, IL;Illinois University Carbondale, IL
Venue:
Proceedings of the 2013 Research in Adaptive and Convergent Systems
Year:
2013

Citing 8
Cited 0

The state of the art in distributed query processing

ACM Computing Surveys (CSUR)
MapReduce: simplified data processing on large clusters

OSDI'04 Proceedings of the 6th conference on Symposium on Opearting Systems Design & Implementation - Volume 6
Pig latin: a not-so-foreign language for data processing

Proceedings of the 2008 ACM SIGMOD international conference on Management of data
Heuristic for resources allocation on utility computing infrastructures

Proceedings of the 6th international workshop on Middleware for grid computing
A Particle Swarm Optimization-Based Heuristic for Scheduling Workflow Applications in Cloud Computing Environments

AINA '10 Proceedings of the 2010 24th IEEE International Conference on Advanced Information Networking and Applications
Profit-Driven Service Request Scheduling in Clouds

CCGRID '10 Proceedings of the 2010 10th IEEE/ACM International Conference on Cluster, Cloud and Grid Computing
Schedule optimization for data processing flows on the cloud

Proceedings of the 2011 ACM SIGMOD International Conference on Management of data
A Fault-Tolerant Workflow Mapping Algorithm under End-to-End Delay Constraint

HPCC '11 Proceedings of the 2011 IEEE International Conference on High Performance Computing and Communications

Quantified Score

Hi-index	0.00

Visualization

Abstract

Cloud computing is emerging as a promising paradigm for large-scale data-intensive queries modeled as complex Directed Acyclic Graph (DAG)-structured dataflows with arbitrary data operators as nodes and producer-consumer interactions as directed edges. The optimization problem of scheduling dataflows on the Cloud is a very complex and challenging task which is similar to query optimization. Optimization must satisfy a variety of objectives and constraints, while taking into account the particular characteristics of the underlying Cloud environment. In addition to achieving minimum completion time, the commercialization of Clouds requires policies to take users' economic concerns as well. In this paper, we formulate scheduling of dataflows onto Cloud resources under the objective of minimizing the completion time under certain budget constraint. A heuristic scheduling algorithm, Layer-oriented Resource Allocation within Budget constraint (LRA-B) is proposed and evaluated. Experiments are conducted on numerous dataflows and Cloud environment configurations, and the overall results are quite promising and indicate the effectiveness of our algorithm.