Experimental evaluation of infiniband transport over local- and wide-area networks

  • Authors:
  • Steven Carter;Makia Minich;Nageswara S. V. Rao

  • Affiliations:
  • Oak Ridge National Laboratory Oak Ridge, TN;Oak Ridge National Laboratory Oak Ridge, TN;Oak Ridge National Laboratory Oak Ridge, TN

  • Venue:
  • SpringSim '07 Proceedings of the 2007 spring simulation multiconference - Volume 2
  • Year:
  • 2007

Quantified Score

Hi-index 0.00

Visualization

Abstract

High-performance file systems that span across supercomputers and storage systems are becoming increasingly important, as these resources are being dispersed both within the enterprise and across the wide-area. Infiniband transport has been proposed as a candidate to provide high-performance access over interconnects primarily within an enterprise. In this paper, we present an experimental study of Infiniband transport over ORNL infrastructure of Cray XT3 supercomputer and cluster storage system for local-area and DOE UltraScience Network (USN) for wide-area. Our results indicate that Infiniband throughput of 7.6Gbps (4x) scales to 8,000 miles with 5% degradation with no customization over 10Gbps wide-area connections. We conducted preliminary performance tests of Lustre file system over these connections, which identified potential parameter and modules optimizations.