High performance wide-area overlay using deadlock-free routing
Proceedings of the 18th ACM international symposium on High performance distributed computing
A scalable high-performance communication library for wide-area environments
GRID '08 Proceedings of the 2008 9th IEEE/ACM International Conference on Grid Computing
| Hi-index | 0.00 |
We propose a connection management scheme that limits the number of inter-cluster connections and forwards messages for processes that cannot communicate directly. We also propose a rank assignment scheme that finds rankprocess mappings with low communication overhead by solving the Quadratic Assignment Problem. Our proposed methods perform locality-aware communication optimizations, and do so without tedious manual configuration by obtaining latency and traffic information from a short profiling run of the environment and the application. Using these methods, we implemented a wide-area-enabled MPI library called MC-MPI, and evaluated its performance by running the NAS Parallel Benchmarks on 256 real nodes distributed across 4 clusters. MC-MPI was able to limit the number of process pairs that established connections to just 10% without suffering a performance penalty. Moreover, MC-MPI was able to find rank assignments that resulted in up to 160% better performance than locality-unaware assignments.