G-Hadoop: MapReduce across distributed data centers for data-intensive computing

  • Authors:

  • Lizhe Wang;Jie Tao;Rajiv Ranjan;Holger Marten;Achim Streit;Jingying Chen;Dan Chen

  • Affiliations:

  • School of Computer, China University of Geosciences, PR China and Center for Earth Observation and Digital Earth, Chinese Academy of Sciences, PR China;Steinbuch Center for Computing, Karlsruhe Institute of Technology, Germany;ICT Centre, CSIRO, Australia;Steinbuch Center for Computing, Karlsruhe Institute of Technology, Germany;Steinbuch Center for Computing, Karlsruhe Institute of Technology, Germany;National Engineering Center for E-Learning, Central China Normal University, PR China;School of Computer, China University of Geosciences, PR China

  • Venue:
  • Future Generation Computer Systems
  • Year:
  • 2013

Quantified Score

Hi-index 0.00

Visualization

Abstract

Recently, the computational requirements for large-scale data-intensive analysis of scientific data have grown significantly. In High Energy Physics (HEP) for example, the Large Hadron Collider (LHC) produced 13 petabytes of data in 2010. This huge amount of data is processed on more than 140 computing centers distributed across 34 countries. The MapReduce paradigm has emerged as a highly successful programming model for large-scale data-intensive computing applications. However, current MapReduce implementations are developed to operate on single cluster environments and cannot be leveraged for large-scale distributed data processing across multiple clusters. On the other hand, workflow systems are used for distributed data processing across data centers. It has been reported that the workflow paradigm has some limitations for distributed data processing, such as reliability and efficiency. In this paper, we present the design and implementation of G-Hadoop, a MapReduce framework that aims to enable large-scale distributed computing across multiple clusters.