Characterization and modeling of PIDX parallel I/O for performance optimization

Authors:
Sidharth Kumar;Avishek Saha;Venkatram Vishwanath;Philip Carns;John A. Schmidt;Giorgio Scorzelli;Hemanth Kolla;Ray Grout;Robert Latham;Robert Ross;Michael E. Papkafa;Jacqueline Chen;Valerio Pascucci
Affiliations:
University of Utah, Salt Lake City, UT;University of Utah, Salt Lake City, UT;Argonne National Laboratory, Argonne, IL;Argonne National Laboratory, Argonne, IL;University of Utah, Salt Lake City, UT;University of Utah, Salt Lake City, UT;Sandia National Laboratories, Livermore, CA;National Renewable Energy Laboratory, Golden, CO;Argonne National Laboratory, Argonne, IL;University of Utah, Salt Lake City, UT;Argonne National Laboratory, Argonne, IL and Northern Illinois University, DeKalb, IL;Sandia National Laboratories, Livermore, CA;University of Utah, Salt Lake City, UT and Pacific Northwest National Laboratory, Richland, WA
Venue:
SC '13 Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis
Year:
2013

Citing 29
Cited 0

Improved parallel I/O via a two-phase run-time access strategy

ACM SIGARCH Computer Architecture News - Special issue on input/output in parallel computer systems
An analytic performance model of disk arrays

SIGMETRICS '93 Proceedings of the 1993 ACM SIGMETRICS conference on Measurement and modeling of computer systems
Global static indexing for real-time exploration of very large regular grids

Proceedings of the 2001 ACM/IEEE conference on Supercomputing
Communication characteristics of large-scale scientific applications for contemporary cluster architectures

Journal of Parallel and Distributed Computing - Special section best papers from the 2002 international parallel and distributed processing symposium
The workload on parallel supercomputers: modeling the characteristics of rigid jobs

Journal of Parallel and Distributed Computing
Convex Optimization

Convex Optimization
Performance Models for Evaluation and Automatic Tuning of Symmetric Sparse Matrix-Vector Multiply

ICPP '04 Proceedings of the 2004 International Conference on Parallel Processing
Predicting Inter-Thread Cache Contention on a Chip Multi-Processor Architecture

HPCA '05 Proceedings of the 11th International Symposium on High-Performance Computer Architecture
Parallel netCDF: A High-Performance Scientific I/O Interface

Proceedings of the 2003 ACM/IEEE conference on Supercomputing
Performance Comparison of MPI Implementations over InfiniBand, Myrinet and Quadrics

Proceedings of the 2003 ACM/IEEE conference on Supercomputing
High performance MPI-2 one-sided communication over InfiniBand

CCGRID '04 Proceedings of the 2004 IEEE International Symposium on Cluster Computing and the Grid
Pattern Recognition and Machine Learning (Information Science and Statistics)

Pattern Recognition and Machine Learning (Information Science and Statistics)
Characterizing the I/O behavior of scientific applications on the Cray XT

PDSW '07 Proceedings of the 2nd international workshop on Petascale data storage: held in conjunction with Supercomputing '07
Characterizing and predicting the I/O performance of HPC applications using a parameterized synthetic benchmark

Proceedings of the 2008 ACM/IEEE conference on Supercomputing
Performance modeling in action: Performance prediction of a Cray XT4 system during upgrade

IPDPS '09 Proceedings of the 2009 IEEE International Symposium on Parallel&Distributed Processing
I/O performance challenges at leadership scale

Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis
Using Subfiling to Improve Programming Flexibility and Performance of Parallel Shared-file I/O

ICPP '09 Proceedings of the 2009 International Conference on Parallel Processing
Addressing shared resource contention in multicore processors via scheduling

Proceedings of the fifteenth edition of ASPLOS on Architectural support for programming languages and operating systems
A case for NUMA-aware contention management on multicore systems

Proceedings of the 19th international conference on Parallel architectures and compilation techniques
Interactive editing of massive imagery made simple: Turning Atlanta into Atlantis

ACM Transactions on Graphics (TOG)
Cuanta: quantifying effects of shared on-chip resource interference for consolidated virtual machines

Proceedings of the 2nd ACM Symposium on Cloud Computing
Topology-aware data movement and staging for I/O acceleration on Blue Gene/P supercomputing systems

Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis
PIDX: Efficient Parallel I/O for Multi-resolution Multi-dimensional Scientific Datasets

CLUSTER '11 Proceedings of the 2011 IEEE International Conference on Cluster Computing
Scikit-learn: Machine Learning in Python

The Journal of Machine Learning Research
Characterizing output bottlenecks in a supercomputer

SC '12 Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis
Efficient data restructuring and aggregation for I/O acceleration in PIDX

SC '12 Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis
A practical method for estimating performance degradation on multicore processors, and its application to HPC workloads

SC '12 Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis
Scalable in situ scientific data encoding for analytical query processing

Proceedings of the 22nd international symposium on High-performance parallel and distributed computing
Abstract: Auto-Tuning of Parallel IO Parameters for HDF5 Applications

SCC '12 Proceedings of the 2012 SC Companion: High Performance Computing, Networking Storage and Analysis

Quantified Score

Hi-index	0.00

Visualization

Abstract

Parallel I/O library performance can vary greatly in response to user-tunable parameter values such as aggregator count, file count, and aggregation strategy. Unfortunately, manual selection of these values is time consuming and dependent on characteristics of the target machine, the underlying file system, and the dataset itself. Some characteristics, such as the amount of memory per core, can also impose hard constraints on the range of viable parameter values. In this work we address these problems by using machine learning techniques to model the performance of the PIDX parallel I/O library and select appropriate tunable parameter values. We characterize both the network and I/O phases of PIDX on a Cray XE6 as well as an IBM Blue Gene/P system. We use the results of this study to develop a machine learning model for parameter space exploration and performance prediction.