Dynamic Memory Adjustment for External Mergesort
VLDB '97 Proceedings of the 23rd International Conference on Very Large Data Bases
Memory resource management in VMware ESX server
ACM SIGOPS Operating Systems Review - OSDI '02: Proceedings of the 5th symposium on Operating systems design and implementation
MapReduce: simplified data processing on large clusters
OSDI'04 Proceedings of the 6th conference on Symposium on Opearting Systems Design & Implementation - Volume 6
Dryad: distributed data-parallel programs from sequential building blocks
Proceedings of the 2nd ACM SIGOPS/EuroSys European Conference on Computer Systems 2007
Dynamic memory balancing for virtual machines
ACM SIGOPS Operating Systems Review
Nephele: efficient parallel data processing in the cloud
Proceedings of the 2nd Workshop on Many-Task Computing on Grids and Supercomputers
Nephele/PACTs: a programming model and execution framework for web-scale analytical processing
Proceedings of the 1st ACM symposium on Cloud computing
Mesos: a platform for fine-grained resource sharing in the data center
Proceedings of the 8th USENIX conference on Networked systems design and implementation
Evaluating Adaptive Compression to Mitigate the Effects of Shared I/O in Clouds
IPDPSW '11 Proceedings of the 2011 IEEE International Symposium on Parallel and Distributed Processing Workshops and PhD Forum
Applications Know Best: Performance-Driven Memory Overcommit with Ginkgo
CLOUDCOM '11 Proceedings of the 2011 IEEE Third International Conference on Cloud Computing Technology and Science
SkewTune: mitigating skew in mapreduce applications
SIGMOD '12 Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data
Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing
NSDI'12 Proceedings of the 9th USENIX conference on Networked Systems Design and Implementation
PACMan: coordinated memory caching for parallel jobs
NSDI'12 Proceedings of the 9th USENIX conference on Networked Systems Design and Implementation
Spinning fast iterative data flows
Proceedings of the VLDB Endowment
MROrchestrator: A Fine-Grained Resource Orchestration Framework for MapReduce Clusters
CLOUD '12 Proceedings of the 2012 IEEE Fifth International Conference on Cloud Computing
Heterogeneity and dynamicity of clouds at scale: Google trace analysis
Proceedings of the Third ACM Symposium on Cloud Computing
| Hi-index | 0.00 |
Leveraging distributed main memory is becoming an increasingly popular approach to speed up large-scale data-intensive cluster applications. However, despite the growing number of possible performance benefits, recent studies indicate that the static resource partitioning among different applications and users in those clusters often leads to severe memory fragmentation, rendering almost half of the available memory resources unusable. This paper therefore proposes to extend the static memory partitioning of current cluster resource managers by a more dynamic scheme which continues to ensure a fair resource distribution among the tenants but allows individual applications to claim spare main memory on a temporary basis. We show that our new approach is a natural fit for many use cases in the big data domain and can significantly improve the memory utilization and processing efficiency.