A long-distance InfiniBand interconnection between two clusters in production use

Authors:
Sabine Richling;Heinz Kredel;Steffen Hau;Hans-Günther Kruse
Affiliations:
University of Heidelberg, Heidelberg, Germany;University of Mannheim, Mannheim, Germany;University of Mannheim, Mannheim, Germany;University of Mannheim, Mannheim, Germany
Venue:
State of the Practice Reports
Year:
2011

Citing 6
Cited 2

Multiprocessor performance

Multiprocessor performance
Sensitivity of parallel applications to large differences in bandwidth and latency in two-layer interconnects

Future Generation Computer Systems
A decentralized strategy for genetic scheduling in heterogeneous environments

Multiagent and Grid Systems - Grid Computing, high performance and distributed applications
Experimental evaluation of infiniband transport over local- and wide-area networks

SpringSim '07 Proceedings of the 2007 spring simulation multiconference - Volume 2
Experimental Analysis of InfiniBand Transport Services on WAN

NAS '08 Proceedings of the 2008 International Conference on Networking, Architecture, and Storage
Operating two InfiniBand grid clusters over 28 km distance

International Journal of Grid and Utility Computing

Performance analysis and prediction for distributed homogeneous clusters

Computer Science - Research and Development
Making the network scalable: inter-subnet routing in infiniband

Euro-Par'13 Proceedings of the 19th international conference on Parallel Processing

Quantified Score

Hi-index	0.00

Visualization

Abstract

We discuss operational and organizational issues of an InfiniBand interconnection between two clusters over a distance of 28 km in day-to-day production use. We describe the setup of hardware and networking components, and the solution of technical integration problems. Then we present solutions for a federated authorization system for the cluster within our two participating universities and other organizational integration problems. Performance measurements for MPI communication and file access to Lustre storage systems are presented. The results and a simple performance model show that MPI performance is intrinsically poor across the long-distance interconnection with limited bandwidth. However, file access and MPI communication among nodes on each side are barely affected by the limitations of the interconnection even at high load. Our organizational and technical setup allows the operation of the two clusters as a single system with lower administration costs and a better load balance than in a disconnected setup.