Optimized InfiniBandTM fat-tree routing for shift all-to-all communication patterns

  • Authors:
  • Eitan Zahavi;Gregory Johnson;Darren J. Kerbyson;Michael Lang

  • Affiliations:
  • Mellanox Technologies Ltd., Sha'ar Yokneam, Yokneam 20692, Israel;Performance and Architecture Laboratory (PAL), Los Alamos National Laboratory, U.S.A.;Performance and Architecture Laboratory (PAL), Los Alamos National Laboratory, U.S.A.;Performance and Architecture Laboratory (PAL), Los Alamos National Laboratory, U.S.A.

  • Venue:
  • Concurrency and Computation: Practice & Experience - International Supercomputing Conference (ISC07)
  • Year:
  • 2010

Quantified Score

Hi-index 0.00

Visualization

Abstract

Clustered systems have become a dominant architecture of scalable high-performance super computers. In these large-scale computers, the network performance and scalability is as critical as the compute-nodes speed. InfiniBandTM has become a commodity networking solution supporting the stringent latency, bandwidth and scalability requirements of these clusters. The network performance is also affected by its topology, packet routing and the communication patterns the distributed application exercises. Fat-trees are the topology structures used for constructing most large clusters as they are scalable, maintain cross-bisectional-bandwidth (CBB), and are practical to build using fixed-arity switches. In this paper, we propose a fat-tree routing algorithm that provides a congestion-free, all-to-all shift pattern leveraging on the InfiniBandTM static routing capability. The algorithm supports partially populated fat-trees built with switches of arbitrary number of ports and CBB ratios. To evaluate the proposed algorithm, detailed switch and host simulation models were developed and multiple fabric topologies were run. The results of these simulations as well as measurements on real clusters show an improvement in all-to-all delay by avoiding congestion on the fabric. Copyright © 2009 John Wiley & Sons, Ltd. The paper was presented in the International Super Computer 2007 conference in Dresden Germany.