Investigating MapReduce framework extensions for efficient processing of geographically scattered datasets

Authors:
Hrishikesh Gadre;Ivan Rodero;Manish Parashar
Affiliations:
Rutgers University, Piscataway, NJ;Rutgers University, Piscataway, NJ;Rutgers University, Piscataway, NJ
Venue:
ACM SIGMETRICS Performance Evaluation Review
Year:
2011

Citing 7
Cited 0

A Conversation with Jim Gray

Queue - Storage
Turning the postal system into a generic digital communication mechanism

Proceedings of the 2004 conference on Applications, technologies, architectures, and protocols for computer communications
An architecture for internet data transfer

NSDI'06 Proceedings of the 3rd conference on Networked Systems Design & Implementation - Volume 3
MapReduce: simplified data processing on large clusters

Communications of the ACM - 50th anniversary issue: 1958 - 2008
Delay scheduling: a simple technique for achieving locality and fairness in cluster scheduling

Proceedings of the 5th European conference on Computer systems
New Algorithms for Planning Bulk Transfer via Internet and Shipping Networks

ICDCS '10 Proceedings of the 2010 IEEE 30th International Conference on Distributed Computing Systems
Improving MapReduce performance in heterogeneous environments

OSDI'08 Proceedings of the 8th USENIX conference on Operating systems design and implementation

Quantified Score

Hi-index	0.00

Visualization

Abstract

In this paper, we investigate real-world scenarios in which MapReduce programming model and specifically Hadoop framework could be used for processing large-scale, geographically scattered datasets. We propose an Adaptive Reduce Task Scheduling (ARTS) algorithm and evaluate it on a distributed Hadoop cluster involving multiple datacenters as well as the on a shared Hadoop cluster. The evaluation demonstrates that the ARTS algorithm outperforms the default Reduce phase scheduling algorithm in Hadoop framework.