Data redistribution using one-sided transfers to in-memory HDF5 files

Authors:
Jerome Soumagne;John Biddiscombe;Aurélien Esnard
Affiliations:
Swiss National Supercomputing Centre, Manno, Switzerland and INRIA Bordeaux Sud-Ouest, Talence, France;Swiss National Supercomputing Centre, Manno, Switzerland;INRIA Bordeaux Sud-Ouest, Talence, France
Venue:
EuroMPI'11 Proceedings of the 18th European MPI Users' Group conference on Recent advances in the message passing interface
Year:
2011

Citing 4
Cited 0

Optimizing noncontiguous accesses in MPI – IO

Parallel Computing
Using MPI-2: Advanced Features of the Message-Passing Interface

Using MPI-2: Advanced Features of the Message-Passing Interface
An HDF5 MPI virtual file driver for parallel in-situ post-processing

EuroMPI'10 Proceedings of the 17th European MPI users' group meeting conference on Recent advances in the message passing interface
The Gemini System Interconnect

HOTI '10 Proceedings of the 2010 18th IEEE Symposium on High Performance Interconnects

Quantified Score

Hi-index	0.00

Visualization

Abstract

Outputs of simulation codes making use of the HDF5 file format are usually and mainly composed of several different attributes and datasets, storing either lightweight pieces of information or containing heavy parts of data. These objects, when written or read through the HDF5 layer, create metadata and data IO operations of different block sizes, which depend on the precision and dimension of the arrays that are being manipulated. By making use of simple block redistribution strategies, we present in this paper a case study showing HDF5 IO performance improvements for "in-memory" files stored in a distributed shared memory buffer using one-sided communications through the HDF5 API.