An HDF5 MPI virtual file driver for parallel in-situ post-processing

Authors:
Jerome Soumagne;John Biddiscombe;Jerry Clarke
Affiliations:
Swiss National Supercomputing Centre, SCR Department, Manno, Switzerland and INRIA Bordeaux Sud-Ouest, HiePACS Research Team, Talence, France;Swiss National Supercomputing Centre, SCR Department, Manno, Switzerland;US Army Research Laboratory, CIS Directorate, Aberdeen Proving Ground, MD
Venue:
EuroMPI'10 Proceedings of the 17th European MPI users' group meeting conference on Recent advances in the message passing interface
Year:
2010

Citing 6
Cited 3

Emulating Shared Memory to Simplify Distributed-Memory Programming

IEEE Computational Science & Engineering
Enhancements to the eXtensible Data Model and Format (XDMF)

HPCMP-UGC '07 Proceedings of the 2007 DoD High Performance Computing Modernization Program Users Group Conference
Investigation of leading HPC I/O performance using a scientific-application derived benchmark

Proceedings of the 2007 ACM/IEEE conference on Supercomputing
DART: a substrate for high speed asynchronous data IO

HPDC '08 Proceedings of the 17th international symposium on High performance distributed computing
Flexible IO and integration for scientific codes through the adaptable IO system (ADIOS)

CLADE '08 Proceedings of the 6th international workshop on Challenges of large applications in distributed environments
DataStager: scalable data staging services for petascale applications

Proceedings of the 18th ACM international symposium on High performance distributed computing

Data redistribution using one-sided transfers to in-memory HDF5 files

EuroMPI'11 Proceedings of the 18th European MPI Users' Group conference on Recent advances in the message passing interface
Parallel computational steering and analysis for HPC applications using a paraview interface and the HDF5 DSM virtual file driver

EG PGV'11 Proceedings of the 11th Eurographics conference on Parallel Graphics and Visualization
Parallel in situ coupling of simulation with a fully featured visualization system

EG PGV'11 Proceedings of the 11th Eurographics conference on Parallel Graphics and Visualization

Quantified Score

Hi-index	0.00

Visualization

Abstract

With simulation codes becoming more powerful, using more and more resources, and producing larger and larger data, monitoring or post-processing simulation data in-situ has obvious advantages over the conventional approach of saving to - and reloading data from - the file system. The time it takes to write and then read the data from disk is a significant bottleneck for both the simulation and subsequent post-processing. In order to be able to post-process data as efficiently as possible with minimal disruption to the simulation itself, we have developed a parallel virtual file driver for the HDF5 library which acts as an MPI-IO virtual file layer, allowing the simulation to write in parallel to remotely located distributed shared memory instead of writing to disk.