Linear clustering of objects with multiple attributes
SIGMOD '90 Proceedings of the 1990 ACM SIGMOD international conference on Management of data
The design and implementation of a log-structured file system
SOSP '91 Proceedings of the thirteenth ACM symposium on Operating systems principles
Improved parallel I/O via a two-phase run-time access strategy
ACM SIGARCH Computer Architecture News - Special issue on input/output in parallel computer systems
Design and Evaluation of primitives for Parallel I/O
Proceedings of the 1993 ACM/IEEE conference on Supercomputing
An extended two-phase method for accessing sections of out-of-core arrays
Scientific Programming
Efficient input and output for scientific simulations
Proceedings of the sixth workshop on I/O in parallel and distributed systems
Improving fine-grained irregular shared-memory benchmarks by data reordering
Proceedings of the 2000 ACM/IEEE conference on Supercomputing
Optimizing noncontiguous accesses in MPI – IO
Parallel Computing
Global static indexing for real-time exploration of very large regular grids
Proceedings of the 2001 ACM/IEEE conference on Supercomputing
R-trees: a dynamic index structure for spatial searching
SIGMOD '84 Proceedings of the 1984 ACM SIGMOD international conference on Management of data
Data Management: NetCDF: an Interface for Scientific Data Access
IEEE Computer Graphics and Applications
Analysis of the Clustering Properties of the Hilbert Space-Filling Curve
IEEE Transactions on Knowledge and Data Engineering
GPFS: A Shared-Disk File System for Large Computing Clusters
FAST '02 Proceedings of the Conference on File and Storage Technologies
HPCN Europe 1998 Proceedings of the International Conference and Exhibition on High-Performance Computing and Networking
Distributed Computing with Load-Managed Active Storage
HPDC '02 Proceedings of the 11th IEEE International Symposium on High Performance Distributed Computing
Parallel netCDF: A High-Performance Scientific I/O Interface
Proceedings of the 2003 ACM/IEEE conference on Supercomputing
Exploiting Lustre File Joining for Effective Collective IO
CCGRID '07 Proceedings of the Seventh IEEE International Symposium on Cluster Computing and the Grid
Heuristic cleaning algorithms in log-structured file systems
TCON'95 Proceedings of the USENIX 1995 Technical Conference Proceedings
PVFS: a parallel file system for linux clusters
ALS'00 Proceedings of the 4th annual Linux Showcase & Conference - Volume 4
Bitmap Index Design Choices and Their Performance Implications
IDEAS '07 Proceedings of the 11th International Database Engineering and Applications Symposium
Scalable performance of the Panasas parallel file system
FAST'08 Proceedings of the 6th USENIX Conference on File and Storage Technologies
ParColl: Partitioned Collective I/O on the Cray XT
ICPP '08 Proceedings of the 2008 37th International Conference on Parallel Processing
DataStager: scalable data staging services for petascale applications
Proceedings of the 18th ACM international symposium on High performance distributed computing
Adaptable, metadata rich IO methods for portable high performance IO
IPDPS '09 Proceedings of the 2009 IEEE International Symposium on Parallel&Distributed Processing
PLFS: a checkpoint filesystem for parallel applications
Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis
I/O performance challenges at leadership scale
Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis
Using Subfiling to Improve Programming Flexibility and Performance of Parallel Shared-file I/O
ICPP '09 Proceedings of the 2009 International Conference on Parallel Processing
...and eat it too: high read performance in write-optimized HPC I/O middleware file formats
Proceedings of the 4th Annual Workshop on Petascale Data Storage
Managing Variability in the IO Performance of Petascale Storage Systems
Proceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis
Six degrees of scientific data: reading patterns for extreme scale science IO
Proceedings of the 20th international symposium on High performance distributed computing
Moving the Code to the Data - Dynamic Code Deployment Using ActiveSpaces
IPDPS '11 Proceedings of the 2011 IEEE International Parallel & Distributed Processing Symposium
EDO: Improving Read Performance for Scientific Applications through Elastic Data Organization
CLUSTER '11 Proceedings of the 2011 IEEE International Conference on Cluster Computing
Extending scalability of collective IO through nessie and staging
Proceedings of the sixth workshop on Parallel Data Storage
| Hi-index | 0.00 |
Near the dawn of the petascale era, IO libraries had reached a stability in their function and data layout with only incremental changes being incorporated. The shift in technology, particularly the scale of parallel file systems and the number of compute processes, prompted revisiting best practices for optimal IO performance. Among other efforts like PLFS, the project that led to ADIOS, the ADaptable IO System, was motivated by both the shift in technology and the historical requirement, for optimal IO performance, to change how simulations performed IO depending on the platform. To solve both issues, the ADIOS team, along with consultation with other leading IO experts, sought to build a new IO platform based on the assumptions inherent in the petascale hardware platforms. This paper helps inform the design of future IO platforms with a discussion of lessons learned as part of the process of designing and building ADIOS.