Investigations on InfiniBand: efficient network buffer utilization at scale

Authors:
Galen M. Shipman;Ron Brightwell;Brian Barrett;Jeffrey M. Squyres;Gil Bloch
Affiliations:
Los Alamos National Laboratory, Los Alamos, NM;Sandia National Laboratories, Albuquerque, NM;Los Alamos National Laboratory, Los Alamos, NM;Cisco, Inc., San Jose, CA;Mellanox Technologies, Santa Clara, CA
Venue:
PVM/MPI'07 Proceedings of the 14th European conference on Recent Advances in Parallel Virtual Machine and Message Passing Interface
Year:
2007

Citing 4
Cited 3

High-performance and scalable MPI over InfiniBand with reduced memory usage: an in-depth performance analysis

Proceedings of the 2006 ACM/IEEE conference on Supercomputing
Infiniband scalability in open MPI

IPDPS'06 Proceedings of the 20th international conference on Parallel and distributed processing
Shared receive queue based scalable MPI design for infiniband clusters

IPDPS'06 Proceedings of the 20th international conference on Parallel and distributed processing
High performance RDMA protocols in HPC

EuroPVM/MPI'06 Proceedings of the 13th European PVM/MPI User's Group conference on Recent advances in parallel virtual machine and message passing interface

Evaluating Sparse Data Storage Techniques for MPI Groups and Communicators

ICCS '08 Proceedings of the 8th international conference on Computational Science, Part I
X-SRQ - Improving Scalability and Performance of Multi-core InfiniBand Clusters

Proceedings of the 15th European PVM/MPI Users' Group Meeting on Recent Advances in Parallel Virtual Machine and Message Passing Interface
Scalable memory registration for high performance networks using helper threads

Proceedings of the 8th ACM International Conference on Computing Frontiers

Quantified Score

Hi-index	0.00

Visualization

Abstract

The default messaging model for the OpenFabrics "Verbs" API is to consume receive buffers in order--regardless of the actual incoming message size--leading to inefficient registered memory usage. For example, many small messages can consume large amounts of registered memory. This paper introduces a new transport protocol in Open MPI implemented using the existing OpenFabrics Verbs API that exhibits efficient registered memory utilization. Several real-world applications were run at scale with the new protocol; results show that global network resource utilization efficiency increases, allowing increased scalability--and larger problem sizes--on clusters which can increase application performance in some cases.