Evaluating I/O characteristics and methods for storing structured scientific data

Authors:
Avery Ching;Alok Choudhary;Wei-keng Liao;Lee Ward;Neil Pundit
Affiliations:
Northwestern University, Department of EECS, Evanston, IL;Northwestern University, Department of EECS, Evanston, IL;Northwestern University, Department of EECS, Evanston, IL;Sandia National Laboratories, Scalable Computer Systems Department, Albuquerque, NM;Sandia National Laboratories, Scalable Computer Systems Department, Albuquerque, NM
Venue:
IPDPS'06 Proceedings of the 20th international conference on Parallel and distributed processing
Year:
2006

Citing 12
Cited 7

Input/output characteristics of scalable parallel applications

Supercomputing '95 Proceedings of the 1995 ACM/IEEE conference on Supercomputing
The Vesta parallel file system

ACM Transactions on Computer Systems (TOCS)
The galley parallel file system

ICS '96 Proceedings of the 10th international conference on Supercomputing
On implementing MPI-IO portably and with high performance

Proceedings of the sixth workshop on I/O in parallel and distributed systems
Diving Deep: Data-Management and Visualization Strategies for Adaptive Mesh Refinement Simulations

Computing in Science and Engineering
GPFS: A Shared-Disk File System for Large Computing Clusters

FAST '02 Proceedings of the Conference on File and Storage Technologies
Noncontiguous I/O Accesses Through MPI-IO

CCGRID '03 Proceedings of the 3st International Symposium on Cluster Computing and the Grid
Data Sieving and Collective I/O in ROMIO

FRONTIERS '99 Proceedings of the The 7th Symposium on the Frontiers of Massively Parallel Computation
Parallel netCDF: A High-Performance Scientific I/O Interface

Proceedings of the 2003 ACM/IEEE conference on Supercomputing
PVFS: a parallel file system for linux clusters

ALS'00 Proceedings of the 4th annual Linux Showcase & Conference - Volume 4
Evaluating structured I/O methods for parallel file systems

International Journal of High Performance Computing and Networking
Driving scientific applications by data in distributed environments

ICCS'03 Proceedings of the 2003 international conference on Computational science

Pattern Matching and I/O Replay for POSIX I/O in Parallel Programs

Euro-Par '09 Proceedings of the 15th International Euro-Par Conference on Parallel Processing
Adaptive multi-level cache allocation in distributed storage architectures

Proceedings of the 24th ACM International Conference on Supercomputing
ParaTrac: a fine-grained profiler for data-intensive workflows

Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing
IOrchestrator: Improving the Performance of Multi-node I/O Systems via Inter-Server Coordination

Proceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis
QoS aware storage cache management in multi-server environments

Proceedings of the 16th ACM symposium on Principles and practice of parallel programming
Virtual I/O caching: dynamic storage cache management for concurrent workloads

Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis
Improving collective I/O performance using pipelined two-phase I/O

Proceedings of the 2012 Symposium on High Performance Computing

Quantified Score

Hi-index	0.00

Visualization

Abstract

Many large-scale scientific simulations generate large, structured multi-dimensional datasets. Data is stored at various intervals on high performance I/O storage systems for checkpointing, post-processing, and visualization. Data storage is very I/O intensive and can dominate the overall running time of an application, depending on the characteristics of the I/O access pattern. Our NCIO benchmark determines how I/O characteristics greatly affect performance (up to 2 orders of magnitude) and provides scientific application developers with guidelines for improvement. In this paper, we examine the impact of various I/O parameters and methods when using the MPI-IO interface to store structured scientific data in an optimized parallel file system.