A MapReduce-supported network structure for data centers

  • Authors:

  • Zeliu Ding;Deke Guo;Xue Liu;Xueshan Luo;Guihai Chen

  • Affiliations:

  • School of Information Systems and Management, National University of Defense Technology, Changsha 410073, China and School of Computer Science, McGill University, Montreal, H3A 2A7, Canada;School of Information Systems and Management, National University of Defense Technology, Changsha 410073, China;School of Computer Science, McGill University, Montreal, H3A 2A7, Canada;School of Information Systems and Management, National University of Defense Technology, Changsha 410073, China;Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, 200240, China

  • Venue:
  • Concurrency and Computation: Practice & Experience
  • Year:
  • 2012

Quantified Score

Hi-index 0.00

Visualization

Abstract

Several novel data center network structures have been proposed to improve the topological properties of data centers. A common characteristic of these structures is that they are designed for supporting general applications and services. Consequently, these structures do not match well with the specific requirements of some dedicated applications. In this paper, we propose a hyper-fat-tree network (HFN): a novel data center structure for MapReduce, a well-known distributed data processing application. HFN possesses the advanced characteristics of BCube as well as fat-tree structures and naturally supports MapReduce. We then address several challenging issues that face HFN in supporting MapReduce. Mathematical analysis and comprehensive evaluation show that HFN possesses excellent properties and is indeed a viable structure for MapReduce in practice. Copyright © 2011 John Wiley & Sons, Ltd.