Extendible hashing—a fast access method for dynamic files
ACM Transactions on Database Systems (TODS)
Parallel I/O for high performance computing
Parallel I/O for high performance computing
GPFS: A Shared-Disk File System for Large Computing Clusters
FAST '02 Proceedings of the Conference on File and Storage Technologies
Flexible IO and integration for scientific codes through the adaptable IO system (ADIOS)
CLADE '08 Proceedings of the 6th international workshop on Challenges of large applications in distributed environments
Proceedings of the 2008 ACM/IEEE conference on Supercomputing
Managing Variability in the IO Performance of Petascale Storage Systems
Proceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis
Design and implementation of parallel file aggregation mechanism
Proceedings of the 1st International Workshop on Runtime and Operating Systems for Supercomputers
A system level view of Petascale I/O on IBM Blue Gene/P
Computer Science - Research and Development
Parallel I/O and the metadata wall
Proceedings of the sixth workshop on Parallel Data Storage
Improving the scalability of performance evaluation tools
PARA'10 Proceedings of the 10th international conference on Applied Parallel and Scientific Computing - Volume 2
Enabling event tracing at leadership-class scale through I/O forwarding middleware
Proceedings of the 21st international symposium on High-Performance Parallel and Distributed Computing
A dynamic and adaptive load balancing strategy for parallel file system with large-scale I/O servers
Journal of Parallel and Distributed Computing
Strategies for real-time event reduction
Euro-Par'12 Proceedings of the 18th international conference on Parallel processing workshops
Proceedings of the Conference on Extreme Science and Engineering Discovery Environment: Gateway to Discovery
Optimizing I/O forwarding techniques for extreme-scale event tracing
Cluster Computing
Hi-index | 0.00 |
Parallel applications often store data in multiple task-local files, for example, to remember checkpoints, to circumvent memory limitations, or to record performance data. When operating at very large processor configurations, such applications often experience scalability limitations when the simultaneous creation of thousands of files causes metadataserver contention or simply when large file counts complicate file management or operations on those files even destabilize the file system. SIONlib is a parallel I/O library that addresses this problem by transparently mapping a large number of task-local files onto a small number of physical files via internal metadata handling and block alignment to ensure high performance. While requiring only minimal source code changes, SIONlib significantly reduces file creation overhead and simplifies file handling without penalizing read and write performance. We evaluate SIONlib's efficiency with up to 288 K tasks and report significant performance improvements in two application scenarios.