Technical perspective: the data center is the computer
Communications of the ACM - 50th anniversary issue: 1958 - 2008
A Dynamic MapReduce Scheduler for Heterogeneous Workloads
GCC '09 Proceedings of the 2009 Eighth International Conference on Grid and Cooperative Computing
Mesos: a platform for fine-grained resource sharing in the data center
Proceedings of the 8th USENIX conference on Networked systems design and implementation
Dominant resource fairness: fair allocation of multiple resource types
Proceedings of the 8th USENIX conference on Networked systems design and implementation
Elastic phoenix: malleable mapreduce for shared-memory systems
NPC'11 Proceedings of the 8th IFIP international conference on Network and parallel computing
Workload Characteristic Oriented Scheduler for MapReduce
ICPADS '12 Proceedings of the 2012 IEEE 18th International Conference on Parallel and Distributed Systems
| Hi-index | 0.00 |
The resource management of a multi-tenant MapReduce cluster can be hard given unpredictable user demands. Conventional resource management scheme would inevitably create a fair amount of spare resource fragments in the system. On the other hand, MapReduce workloads are prone to have a bottleneck stage in the execution pipeline. To address these two issues under a coherent framework, this paper presents Clotho, a MapReduce workload and runtime co-design that can opportunistically utilize the spare resource fragments in the system to alleviate the bottleneck stage of MapReduce workloads while honoring the SLAs of existing systems. We describe the design and the implementation of Clotho, evaluate it with benchmarks drawn from real MapReduce applications, and demonstrate that it can effectively utilize the spare CPU resource fragments and meanwhile improve the performance of user programs if there is potential to speed up the bottleneck stage of the entire MapReduce execution pipeline.