Eddies: continuously adaptive query processing
SIGMOD '00 Proceedings of the 2000 ACM SIGMOD international conference on Management of data
Algorithms, games, and the internet
STOC '01 Proceedings of the thirty-third annual ACM symposium on Theory of computing
Access path selection in a relational database management system
SIGMOD '79 Proceedings of the 1979 ACM SIGMOD international conference on Management of data
Mariposa: a wide-area distributed database system
The VLDB Journal — The International Journal on Very Large Data Bases
User-Centric Performance Analysis of Market-Based Cluster Batch Schedulers
CCGRID '02 Proceedings of the 2nd IEEE/ACM International Symposium on Cluster Computing and the Grid
Flux: An Adaptive Partitioning Operator for Continuous Query Systems
Flux: An Adaptive Partitioning Operator for Continuous Query Systems
Market-based Proportional Resource Sharing for Clusters
Market-based Proportional Resource Sharing for Clusters
Xen and the art of virtualization
SOSP '03 Proceedings of the nineteenth ACM symposium on Operating systems principles
Balancing Risk and Reward in a Market-Based Task Service
HPDC '04 Proceedings of the 13th IEEE International Symposium on High Performance Distributed Computing
A price-anticipating resource allocation mechanism for distributed shared clusters
Proceedings of the 6th ACM conference on Electronic commerce
Profitable services in an uncertain world
SC '05 Proceedings of the 2005 ACM/IEEE conference on Supercomputing
Tycoon: An implementation of a distributed, market-based resource allocation system
Multiagent and Grid Systems
Interpreting the data: Parallel analysis with Sawzall
Scientific Programming - Dynamic Grids and Worldwide Computing
MapReduce: simplified data processing on large clusters
OSDI'04 Proceedings of the 6th conference on Symposium on Opearting Systems Design & Implementation - Volume 6
Mirage: a microeconomic resource allocation system for sensornet testbeds
EmNets '05 Proceedings of the 2nd IEEE workshop on Embedded Networked Sensors
Adaptive and scalable comparison scheduling
Proceedings of the 2007 ACM SIGMETRICS international conference on Measurement and modeling of computer systems
Lottery scheduling: flexible proportional-share resource management
OSDI '94 Proceedings of the 1st USENIX conference on Operating Systems Design and Implementation
Argon: performance insulation for shared storage servers
FAST '07 Proceedings of the 5th USENIX conference on File and Storage Technologies
A statistical approach to risk mitigation in computational markets
Proceedings of the 16th international symposium on High performance distributed computing
Dryad: distributed data-parallel programs from sequential building blocks
Proceedings of the 2nd ACM SIGOPS/EuroSys European Conference on Computer Systems 2007
Evaluating MapReduce for Multi-core and Multiprocessor Systems
HPCA '07 Proceedings of the 2007 IEEE 13th International Symposium on High Performance Computer Architecture
Scheduling despite inexact job-size information
SIGMETRICS '08 Proceedings of the 2008 ACM SIGMETRICS international conference on Measurement and modeling of computer systems
Pig latin: a not-so-foreign language for data processing
Proceedings of the 2008 ACM SIGMOD international conference on Management of data
Fully distributed EM for very large datasets
Proceedings of the 25th international conference on Machine learning
Automatic optimization of parallel dataflow programs
ATC'08 USENIX 2008 Annual Technical Conference on Annual Technical Conference
Oivos: Simple and Efficient Distributed Data Processing
HPCC '08 Proceedings of the 2008 10th IEEE International Conference on High Performance Computing and Communications
Mars: a MapReduce framework on graphics processors
Proceedings of the 17th international conference on Parallel architectures and compilation techniques
Ad-hoc data processing in the cloud
Proceedings of the VLDB Endowment
Improving MapReduce performance in heterogeneous environments
OSDI'08 Proceedings of the 8th USENIX conference on Operating systems design and implementation
The efficiency and fairness of a fixed budget resource allocation game
ICALP'05 Proceedings of the 32nd international conference on Automata, Languages and Programming
What's inside the Cloud? An architectural map of the Cloud landscape
CLOUD '09 Proceedings of the 2009 ICSE Workshop on Software Engineering Challenges of Cloud Computing
Topology-aware resource allocation for data-intensive workloads
Proceedings of the first ACM asia-pacific workshop on Workshop on systems
Reining in the outliers in map-reduce clusters using Mantri
OSDI'10 Proceedings of the 9th USENIX conference on Operating systems design and implementation
Topology-aware resource allocation for data-intensive workloads
ACM SIGCOMM Computer Communication Review
Dynamic proportional share scheduling in Hadoop
JSSPP'10 Proceedings of the 15th international conference on Job scheduling strategies for parallel processing
On scheduling in map-reduce and flow-shops
Proceedings of the twenty-third annual ACM symposium on Parallelism in algorithms and architectures
Exploring MapReduce efficiency with highly-distributed data
Proceedings of the second international workshop on MapReduce and its applications
Enhancement of Xen's scheduler for MapReduce workloads
Proceedings of the 20th international symposium on High performance distributed computing
FLEX: a slot allocation scheduling optimizer for MapReduce workloads
Proceedings of the ACM/IFIP/USENIX 11th International Conference on Middleware
Purlieus: locality-aware resource allocation for MapReduce in a cloud
Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis
Performance evaluation of MapReduce using full virtualisation on a departmental cloud
International Journal of Applied Mathematics and Computer Science - SPECIAL SECTION: Efficient Resource Management for Grid-Enabled Applications
Thread Tranquilizer: Dynamically reducing performance variation
ACM Transactions on Architecture and Code Optimization (TACO) - HIPEAC Papers
Jockey: guaranteed job latency in data parallel clusters
Proceedings of the 7th ACM european conference on Computer Systems
Panacea: towards holistic optimization of MapReduce applications
Proceedings of the Tenth International Symposium on Code Generation and Optimization
A MapReduce-supported network structure for data centers
Concurrency and Computation: Practice & Experience
A Hybrid Scheduling Algorithm for Data Intensive Workloads in a MapReduce Environment
UCC '12 Proceedings of the 2012 IEEE/ACM Fifth International Conference on Utility and Cloud Computing
SemanMR: big data processing framework based on semantics
Proceedings of the Fourth Asia-Pacific Symposium on Internetware
Interference and locality-aware task scheduling for MapReduce applications in virtual clusters
Proceedings of the 22nd international symposium on High-performance parallel and distributed computing
Speeding-up codon analysis on the cloud with local MapReduce aggregation
Information Sciences: an International Journal
| Hi-index | 0.00 |
We present a system for allocating resources in shared data and compute clusters that improves MapReduce job scheduling in three ways. First, the system uses regulated and user-assigned priorities to offer different service levels to jobs and users over time. Second, the system dynamically adjusts resource allocations to fit the requirements of different job stages. Finally, the system automatically detects and eliminates bottlenecks within a job. We show experimentally using real applications that users can optimize not only job execution time but also the cost-benefit ratio or prioritization efficiency of a job using these three strategies. Our approach relies on a proportional share mechanism that continuously allocates virtual machine resources. Our experimental results show a 11-31% improvement in completion time and 4-187% improvement in prioritization efficiency for different classes of MapReduce jobs. We further show that delay intolerant users gain even more from our system.