An extended two-phase method for accessing sections of out-of-core arrays
Scientific Programming
On implementing MPI-IO portably and with high performance
Proceedings of the sixth workshop on I/O in parallel and distributed systems
Using MPI-2: Advanced Features of the Message Passing Interface
Using MPI-2: Advanced Features of the Message Passing Interface
A case study in application I/O on Linux clusters
Proceedings of the 2001 ACM/IEEE conference on Supercomputing
GPFS: A Shared-Disk File System for Large Computing Clusters
FAST '02 Proceedings of the Conference on File and Storage Technologies
I/O Analysis and Optimization for an AMR Cosmology Application
CLUSTER '02 Proceedings of the IEEE International Conference on Cluster Computing
An Abstract-Device Interface for Implementing Portable Parallel-I/O Interfaces
FRONTIERS '96 Proceedings of the 6th Symposium on the Frontiers of Massively Parallel Computation
Data Sieving and Collective I/O in ROMIO
FRONTIERS '99 Proceedings of the The 7th Symposium on the Frontiers of Massively Parallel Computation
Interactive Exploration of Large Remote Micro-CT Scans
VIS '04 Proceedings of the conference on Visualization '04
A Parallel Visualization Pipeline for Terascale Earthquake Simulations
Proceedings of the 2004 ACM/IEEE conference on Supercomputing
High-Level Buffering for Hiding Periodic Output Cost in Scientific Simulations
IEEE Transactions on Parallel and Distributed Systems
Scaling Properties of Common Statistical Operators for Gridded Datasets
International Journal of High Performance Computing Applications
Efficient Parallel I/O in Community Atmosphere Model (CAM)
International Journal of High Performance Computing Applications
Noncontiguous locking techniques for parallel file systems
Proceedings of the 2007 ACM/IEEE conference on Supercomputing
Scaling climate simulation applications on the IBM Blue Gene/L system
IBM Journal of Research and Development
Toward automatic parallelization of spatial computation for computing clusters
HPDC '08 Proceedings of the 17th international symposium on High performance distributed computing
Implementing a Parallel NetCDF Interface for Seamless Remote I/O Using Multi-dimensional Data
High Performance Computing for Computational Science - VECPAR 2008
Interfaces for parallel numerical linear algebra libraries in high level languages
Advances in Engineering Software
Querying for Feature Extraction and Visualization in Climate Modeling
ICCS 2009 Proceedings of the 9th International Conference on Computational Science
MPISec I/O: Providing Data Confidentiality in MPI-I/O
CCGRID '09 Proceedings of the 2009 9th IEEE/ACM International Symposium on Cluster Computing and the Grid
PyACTS: a python based interface to ACTS tools and parallel scientific applications
International Journal of Parallel Programming
Processing MPI Datatypes Outside MPI
Proceedings of the 16th European PVM/MPI Users' Group Meeting on Recent Advances in Parallel Virtual Machine and Message Passing Interface
Parallelization and I/O optimization for a 3-D plasma simulation code
ICCOMP'09 Proceedings of the WSEAES 13th international conference on Computers
On the Need for a Consortium of Capability Centers
International Journal of High Performance Computing Applications
PyPnetCDF: A high level framework for parallel access to netCDF files
Advances in Engineering Software
Terascale data organization for discovering multivariate climatic trends
Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis
PLFS: a checkpoint filesystem for parallel applications
Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis
Armazenamento distribuído de imagens médicas DICOM no formato de dados HDF5
Proceedings of the 14th Brazilian Symposium on Multimedia and the Web
pNFS, POSIX, and MPI-IO: a tale of three semantics
Proceedings of the 4th Annual Workshop on Petascale Data Storage
Fusing data management services with file systems
Proceedings of the 4th Annual Workshop on Petascale Data Storage
Implementation and Evaluation of File Write-Back and Prefetching for MPI-IO Over GPFS
International Journal of High Performance Computing Applications
A Scalable Message Passing Interface Implementation of an Ad-Hoc Parallel I/o system
International Journal of High Performance Computing Applications
A study of real world I/O performance in parallel scientific computing
PARA'06 Proceedings of the 8th international conference on Applied parallel computing: state of the art in scientific computing
MRAP: a novel MapReduce-based framework to support HPC analytics applications with access patterns
Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing
Automated tracing of I/O stack
EuroMPI'10 Proceedings of the 17th European MPI users' group meeting conference on Recent advances in the message passing interface
MPI-IO/L: efficient remote I/O for MPI-IO via logistical networking
IPDPS'06 Proceedings of the 20th international conference on Parallel and distributed processing
Evaluating I/O characteristics and methods for storing structured scientific data
IPDPS'06 Proceedings of the 20th international conference on Parallel and distributed processing
Visual analysis of I/O system behavior for high-end computing
Proceedings of the third international workshop on Large-scale system and application performance
Six degrees of scientific data: reading patterns for extreme scale science IO
Proceedings of the 20th international symposium on High performance distributed computing
Improving the average response time in collective I/O
EuroMPI'11 Proceedings of the 18th European MPI Users' Group conference on Recent advances in the message passing interface
Topology-aware data movement and staging for I/O acceleration on Blue Gene/P supercomputing systems
Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis
Virtual I/O caching: dynamic storage cache management for concurrent workloads
Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis
Improving the efficiency of subset queries on raster images
Proceedings of the ACM SIGSPATIAL Second International Workshop on High Performance and Distributed Geographic Information Systems
Efficient data IO for a Parallel Global Cloud Resolving Model
Environmental Modelling & Software
Effective seamless remote MPI-I/O operations with derived data types using PVFS2
EuroPVM/MPI'06 Proceedings of the 13th European PVM/MPI User's Group conference on Recent advances in parallel virtual machine and message passing interface
On the usability of high-level parallel IO in unstructured grid simulations
EuroPVM/MPI'06 Proceedings of the 13th European PVM/MPI User's Group conference on Recent advances in parallel virtual machine and message passing interface
Towards scalable array-oriented active storage: the pyramid approach
ACM SIGOPS Operating Systems Review
An application-level parallel I/O library for Earth system models
International Journal of High Performance Computing Applications
Extending scalability of collective IO through nessie and staging
Proceedings of the sixth workshop on Parallel Data Storage
Light-Weight parallel i/o analysis at scale
EPEW'11 Proceedings of the 8th European conference on Computer Performance Engineering
Bridging HPC and grid file i/o with IOFSL
PARA'10 Proceedings of the 10th international conference on Applied Parallel and Scientific Computing - Volume 2
ISOBAR hybrid compression-I/O interleaving for large-scale parallel I/O optimization
Proceedings of the 21st international symposium on High-Performance Parallel and Distributed Computing
Supporting User-Defined Subsetting and Aggregation over Parallel NetCDF Datasets
CCGRID '12 Proceedings of the 2012 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (ccgrid 2012)
Proceedings of the 2nd International Workshop on Runtime and Operating Systems for Supercomputers
Specification and performance evaluation of parallel i/o interfaces for OpenMP
IWOMP'12 Proceedings of the 8th international conference on OpenMP in a Heterogeneous World
I/O strategies for parallel rendering of large time-varying volume data
EG PGV'04 Proceedings of the 5th Eurographics conference on Parallel Graphics and Visualization
Efficient I/O for parallel visualization
EG PGV'11 Proceedings of the 11th Eurographics conference on Parallel Graphics and Visualization
A divide and conquer strategy for scaling weather simulations with multiple regions of interest
SC '12 Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis
Byte-precision level of detail processing for variable precision analytics
SC '12 Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis
Efficient data restructuring and aggregation for I/O acceleration in PIDX
SC '12 Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis
Parallel I/O, analysis, and visualization of a trillion particle simulation
SC '12 Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis
Transparent log-based data storage in MPI-IO applications
PVM/MPI'07 Proceedings of the 14th European conference on Recent Advances in Parallel Virtual Machine and Message Passing Interface
On the usability of the MPI shared file pointer routines
EuroMPI'12 Proceedings of the 19th European conference on Recent Advances in the Message Passing Interface
Optimizing fastquery performance on lustre file system
Proceedings of the 25th International Conference on Scientific and Statistical Database Management
Improving collective I/O performance by pipelining request aggregation and file access
Proceedings of the 20th European MPI Users' Group Meeting
Insights for exascale IO APIs from building a petascale IO API
SC '13 Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis
Characterization and modeling of PIDX parallel I/O for performance optimization
SC '13 Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis
11 PFLOP/s simulations of cloud cavitation collapse
SC '13 Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis
On-demand unstructured mesh translation for reducing memory pressure during in situ analysis
UltraVis '13 Proceedings of the 8th International Workshop on Ultrascale Visualization
Asynchronous object storage with QoS for scientific and commercial big data
PDSW '13 Proceedings of the 8th Parallel Data Storage Workshop
GeoMix: scalable geoscientific array data management
Proceedings of the Industrial Track of the 13th ACM/IFIP/USENIX International Middleware Conference
Fluid mechanics based classification of the respiratory efficiency of several nasal cavities
Computers in Biology and Medicine
A divide and conquer strategy for scaling weather simulations with multiple regions of interest
Scientific Programming - Selected Papers from Super Computing 2012
| Hi-index | 0.00 |
Dataset storage, exchange, and access play a critical role in scientific applications. For such purposes netCDF serves as a portable, efficient file format and programming interface, which is popular in numerous scientific application domains. However, the original interface does not provide an efficient mechanism for parallel data storage and access. In this work, we present a new parallel interface for writing and reading netCDF datasets. This interface is derived with minimal changes from the serial netCDF interface but defines semantics for parallel access and is tailored for high performance. The underlying parallel I/O is achieved through MPI-IO, allowing for substantial performance gains through the use of collective I/O optimizations. We compare the implementation strategies and performance with HDF5. Our tests indicate programming convenience and significant I/O performance improvement with this parallel netCDF (PnetCDF) interface.