VMR: volunteer MapReduce over the large scale internet

Authors:
Fernando Costa;Luís Veiga;Paulo Ferreira
Affiliations:
Technical University of Lisbon, Lisboa, Portugal;Technical University of Lisbon, Lisboa, Portugal;Technical University of Lisbon, Lisboa, Portugal
Venue:
Proceedings of the 10th International Workshop on Middleware for Grids, Clouds and e-Science
Year:
2012

Citing 9
Cited 0

SETI@home: an experiment in public-resource computing

Communications of the ACM
PlanetLab: an overlay testbed for broad-coverage services

ACM SIGCOMM Computer Communication Review
BOINC: A System for Public-Resource Computing and Storage

GRID '04 Proceedings of the 5th IEEE/ACM International Workshop on Grid Computing
Computing on large-scale distributed systems: Xtrem Web architecture, programming models, security, tests and convergence with grid

Future Generation Computer Systems - Special issue: P2P computing and interaction with grids
MapReduce: simplified data processing on large clusters

Communications of the ACM - 50th anniversary issue: 1958 - 2008
BitDew: a programmable environment for large-scale data management and distribution

Proceedings of the 2008 ACM/IEEE conference on Supercomputing
Cost-benefit analysis of Cloud Computing versus desktop grids

IPDPS '09 Proceedings of the 2009 IEEE International Symposium on Parallel&Distributed Processing
MOON: MapReduce On Opportunistic eNvironments

Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing
Towards MapReduce for Desktop Grid Computing

3PGCIC '10 Proceedings of the 2010 International Conference on P2P, Parallel, Grid, Cloud and Internet Computing

Quantified Score

Hi-index	0.00

Visualization

Abstract

Volunteer Computing systems (VC) harness computing resources of machines from around the world to perform distributed independent tasks. Existing infrastructures follow a master/worker model, with a centralized architecture, which limits the scalability of the solution given its dependence on the server. We intend to create a distributed model, in order to improve performance and reduce the burden on the server. In this paper we present VMR, a VC system able to run MapReduce applications on top of volunteer resources, over the large scale Internet. We describe VMR's architecture and evaluate its performance by executing several MapReduce applications on a wide area testbed. Our results show that VMR successfully runs MapReduce tasks over the Internet. When compared to an unmodified VC system, VMR obtains a performance increase of over 60% in application turnaround time, while reducing the bandwidth use by an order of magnitude.