Efficient data restructuring and aggregation for I/O acceleration in PIDX

Authors:
Sidharth Kumar;Venkatram Vishwanath;Philip Carns;Joshua A. Levine;Robert Latham;Giorgio Scorzelli;Hemanth Kolla;Ray Grout;Robert Ross;Michael E. Papka;Jacqueline Chen;Valerio Pascucci
Affiliations:
University of Utah, Salt Lake City, UT;Argonne National Laboratory, Argonne, IL;Argonne National Laboratory, Argonne, IL;University of Utah, Salt Lake City, UT;Argonne National Laboratory, Argonne, IL;University of Utah, Salt Lake City, UT;Sandia National Laboratories, Livermore, CA;National Renewable Energy Laboratory, Golden, CO;Argonne National Laboratory, Argonne, IL;Argonne National Laboratory, Argonne, IL;Sandia National Laboratories, Livermore, CA;University of Utah, Salt Lake City, UT
Venue:
SC '12 Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis
Year:
2012

Citing 12
Cited 2

Improved parallel I/O via a two-phase run-time access strategy

ACM SIGARCH Computer Architecture News - Special issue on input/output in parallel computer systems
Global static indexing for real-time exploration of very large regular grids

Proceedings of the 2001 ACM/IEEE conference on Supercomputing
GPFS: A Shared-Disk File System for Large Computing Clusters

FAST '02 Proceedings of the Conference on File and Storage Technologies
Real-time monitoring of large scientific simulations

Proceedings of the 2003 ACM symposium on Applied computing
Parallel netCDF: A High-Performance Scientific I/O Interface

Proceedings of the 2003 ACM/IEEE conference on Supercomputing
Flexible IO and integration for scientific codes through the adaptable IO system (ADIOS)

CLADE '08 Proceedings of the 6th international workshop on Challenges of large applications in distributed environments
Characterizing and predicting the I/O performance of HPC applications using a parameterized synthetic benchmark

Proceedings of the 2008 ACM/IEEE conference on Supercomputing
Using Subfiling to Improve Programming Flexibility and Performance of Parallel Shared-file I/O

ICPP '09 Proceedings of the 2009 International Conference on Parallel Processing
Interactive editing of massive imagery made simple: Turning Atlanta into Atlantis

ACM Transactions on Graphics (TOG)
PIDX: Efficient Parallel I/O for Multi-resolution Multi-dimensional Scientific Datasets

CLUSTER '11 Proceedings of the 2011 IEEE International Conference on Cluster Computing
Efficient data IO for a Parallel Global Cloud Resolving Model

Environmental Modelling & Software
A multiresolution volume rendering framework for large-scale time-varying data visualization

VG'05 Proceedings of the Fourth Eurographics / IEEE VGTC conference on Volume Graphics

Scalable in situ scientific data encoding for analytical query processing

Proceedings of the 22nd international symposium on High-performance parallel and distributed computing
Characterization and modeling of PIDX parallel I/O for performance optimization

SC '13 Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis

Quantified Score

Hi-index	0.00

Visualization

Abstract

Hierarchical, multiresolution data representations enable interactive analysis and visualization of large-scale simulations. One promising application of these techniques is to store high performance computing simulation output in a hierarchical Z (HZ) ordering that translates data from a Cartesian coordinate scheme to a one-dimensional array ordered by locality at different resolution levels. However, when the dimensions of the simulation data are not an even power of 2, parallel HZ ordering produces sparse memory and network access patterns that inhibit I/O performance. This work presents a new technique for parallel HZ ordering of simulation datasets that restructures simulation data into large (power of 2) blocks to facilitate efficient I/O aggregation. We perform both weak and strong scaling experiments using the S3D combustion application on both Cray-XE6 (65,536 cores) and IBM Blue Gene/P (131,072 cores) platforms. We demonstrate that data can be written in hierarchical, multiresolution format with performance competitive to that of native data-ordering methods.