High performance MPI design using unreliable datagram for ultra-scale InfiniBand clusters
Proceedings of the 21st annual international conference on Supercomputing
Proceedings of the 22nd annual international conference on Supercomputing
Design optimization of a highly parallel InfiniBand host channel adapter
Proceedings of the 4th ACM/IEEE Symposium on Architectures for Networking and Communications Systems
Scalable memory registration for high performance networks using helper threads
Proceedings of the 8th ACM International Conference on Computing Frontiers
| Hi-index | 0.00 |
InfiniBand is emerging as a high-performance interconnect. It is gaining popularity because of its high performance and open standard. Recently, PCI-Express, which is the third generation high-performance I/O bus used to interconnect peripheral devices, has been released. The third generation of InfiniBand adapters allow applications to take advantage of PCI-Express. PCI-Express offers very low latency access of the host memory by network interface cards (NICs). Earlier generation InfiniBand adapters used to have an external DIMM attached as local NIC memory. This memory was used to store internal information. This memory increases the overall cost of the NIC. In this paper we design experiments, analyze the performance of various communication patterns and end applications on PCI-Express based systems, whose adapters can be chosen to run with or without local NIC memory. Our investigations reveal that on these systems, the memory fetch latency is the same for both local NIC memory and Host memory. Under heavy I/O bus usage, the latency of a scatter operation increased only by 10% and only for message sizes 1B - 4KB. These memory-less adapters allow more efficientuse of overall system memory and show practically no performance impact (less than 0.1%) for the NAS Parallel Benchmarks on 8 processes. These results indicate that memory-less network adapters can benefit next generation InfiniBand systems.