Scientific workflow management and the Kepler system: Research Articles
Concurrency and Computation: Practice & Experience - Workflow in Grid Systems
Proceedings of the 2006 ACM/IEEE conference on Supercomputing
Pegasus: A framework for mapping complex scientific workflows onto distributed systems
Scientific Programming
LIVE data workspace: A flexible, dynamic and extensible platform for petascale applications
CLUSTER '07 Proceedings of the 2007 IEEE International Conference on Cluster Computing
DataStager: scalable data staging services for petascale applications
Proceedings of the 18th ACM international symposium on High performance distributed computing
Abstract storage: moving file format-specific abstractions intopetabyte-scale storage systems
Proceedings of the second international workshop on Data-aware distributed computing
Event-based systems: opportunities and challenges at exascale
Proceedings of the Third ACM International Conference on Distributed Event-Based Systems
Plasma fusion code coupling using scalable I/O services and scientific workflows
Proceedings of the 4th Workshop on Workflows in Support of Large-Scale Science
Scalable massively parallel I/O to task-local files
Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis
PLFS: a checkpoint filesystem for parallel applications
Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis
Data layout optimization for petascale file systems
Proceedings of the 4th Annual Workshop on Petascale Data Storage
...and eat it too: high read performance in write-optimized HPC I/O middleware file formats
Proceedings of the 4th Annual Workshop on Petascale Data Storage
DataStager: scalable data staging services for petascale applications
Cluster Computing
A layout-aware optimization strategy for collective I/O
Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing
Accelerating parallel analysis of scientific simulation data via Zazen
FAST'10 Proceedings of the 8th USENIX conference on File and storage technologies
Managing Variability in the IO Performance of Petascale Storage Systems
Proceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis
An HDF5 MPI virtual file driver for parallel in-situ post-processing
EuroMPI'10 Proceedings of the 17th European MPI users' group meeting conference on Recent advances in the message passing interface
Exploiting Latent I/O Asynchrony in Petascale Science Applications
International Journal of High Performance Computing Applications
Six degrees of scientific data: reading patterns for extreme scale science IO
Proceedings of the 20th international symposium on High performance distributed computing
Efficient data IO for a Parallel Global Cloud Resolving Model
Environmental Modelling & Software
Towards scalable array-oriented active storage: the pyramid approach
ACM SIGOPS Operating Systems Review
Extending scalability of collective IO through nessie and staging
Proceedings of the sixth workshop on Parallel Data Storage
Enabling event tracing at leadership-class scale through I/O forwarding middleware
Proceedings of the 21st international symposium on High-Performance Parallel and Distributed Computing
ISOBAR hybrid compression-I/O interleaving for large-scale parallel I/O optimization
Proceedings of the 21st international symposium on High-Performance Parallel and Distributed Computing
Efficient I/O for parallel visualization
EG PGV'11 Proceedings of the 11th Eurographics conference on Parallel Graphics and Visualization
EG PGV'11 Proceedings of the 11th Eurographics conference on Parallel Graphics and Visualization
Byte-precision level of detail processing for variable precision analytics
SC '12 Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis
Efficient data restructuring and aggregation for I/O acceleration in PIDX
SC '12 Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis
Parallel I/O, analysis, and visualization of a trillion particle simulation
SC '12 Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis
Towards an energy-aware scientific I/O interface
Computer Science - Research and Development
I/O acceleration with pattern detection
Proceedings of the 22nd international symposium on High-performance parallel and distributed computing
Memory-conscious collective I/O for extreme scale HPC systems
Proceedings of the 3rd International Workshop on Runtime and Operating Systems for Supercomputers
A Maya use case: adaptable scientific workflows with ADIOS for general relativistic astrophysics
Proceedings of the Conference on Extreme Science and Engineering Discovery Environment: Gateway to Discovery
Optimizing fastquery performance on lustre file system
Proceedings of the 25th International Conference on Scientific and Statistical Database Management
11 PFLOP/s simulations of cloud cavitation collapse
SC '13 Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis
Feature-based analysis of large-scale spatio-temporal sensor data on hybrid architectures
International Journal of High Performance Computing Applications
On-demand unstructured mesh translation for reducing memory pressure during in situ analysis
UltraVis '13 Proceedings of the 8th International Workshop on Ultrascale Visualization
Optimizing I/O forwarding techniques for extreme-scale event tracing
Cluster Computing
| Hi-index | 0.00 |
Scientific codes are all subject to variation in performance depending on the runtime platform and/or configuration, the output writing API employed, and the file system for output. Since changing the IO routines to match the optimal or desired configuration for a given system can be costly in terms of human time and machine resources, the Adaptable IO System provides an API nearly as simple as POSIX IO that also provides developers with the flexibility of selection the optimal IO routines for a given platform, without recompilation. As a side effect, we also gain the ability to transparently integrate more tightly with workflow systems like Kepler and Pegasus and visualization systems like Visit with no runtime impact. We achieve this through our library of highly tuned IO routines and other transport methods selected and configured in an XML file read only at startup. ADIOS-based IO has demonstrated high levels of performance and scalability. For example, we have achieved 20 GB/sec write performance using GTC on the Jaguar Cray XT4 system at Oak Ridge National Labs (about 50\% of peak performance). We can change GTC output among MPI-IO synchronous, MPI-IO collective, POSIX IO, no IO (for baseline testing), asynchronous IO using the Georgia Tech DataTap system, and Visit directly for in situ visualization with no changes to the source code. We designed this initial version of ADIOS based on the data requirements of 7 major scientific codes (GTC, Chimera, GTS, XGC1, XGC0, FLASH, and S3D) and have successfully adapted all of them to use ADIOS for all of their IO needs.