Operating two InfiniBand grid clusters over 28 km distance

Authors:
Sabine Richling;Steffen Hau;Heinz Kredel;Hans-Gunther Kruse
Affiliations:
IT Center, University of Heidelberg, 69120 Heidelberg, Germany.;IT Center, University of Mannheim, 68131 Mannheim, Germany.;IT Center, University of Mannheim, 68131 Mannheim, Germany.;IT Center, University of Mannheim, 68131 Mannheim, Germany
Venue:
International Journal of Grid and Utility Computing
Year:
2011

Citing 7
Cited 2

Multiprocessor performance

Multiprocessor performance
Sensitivity of parallel applications to large differences in bandwidth and latency in two-layer interconnects

Future Generation Computer Systems
Computer Architecture: A Quantitative Approach

Computer Architecture: A Quantitative Approach
A decentralized strategy for genetic scheduling in heterogeneous environments

Multiagent and Grid Systems - Grid Computing, high performance and distributed applications
Experimental evaluation of infiniband transport over local- and wide-area networks

SpringSim '07 Proceedings of the 2007 spring simulation multiconference - Volume 2
Experimental Analysis of InfiniBand Transport Services on WAN

NAS '08 Proceedings of the 2008 International Conference on Networking, Architecture, and Storage
Computer Networks, Fifth Edition: A Systems Approach

Computer Networks, Fifth Edition: A Systems Approach

A long-distance InfiniBand interconnection between two clusters in production use

State of the Practice Reports
Performance analysis and prediction for distributed homogeneous clusters

Computer Science - Research and Development

Quantified Score

Hi-index	0.00

Visualization

Abstract

This paper considers an InfiniBand connection between two bwGRiD clusters over a distance of 28 km in day-to-day production use. We discuss the hardware set-up of InfiniBand messages converted and transported over a fibre optic connection. The two clusters can be operated as single system image, the batch system will enforce that all nodes for a job are allocated on one side of the cluster. This is to optimise MPI performance, which would not be sufficient for communication between nodes on opposite sides of the 28 km connection. We report on the successful solution of all technical and organisational integration hurdles. By a simple performance model we discuss the anticipated costs for a doubling in communication performance.