Architectural requirements of parallel scientific applications with explicit communication
ISCA '93 Proceedings of the 20th annual international symposium on computer architecture
The high performance storage system
Proceedings of the 1993 ACM/IEEE conference on Supercomputing
Extensible file system (ELFS): an object-oriented approach to high performance file I/O
OOPSLA '94 Proceedings of the ninth annual conference on Object-oriented programming systems, language, and applications
Parallel file systems for the IBM SP computers
IBM Systems Journal
Server-directed collective I/O in Panda
Supercomputing '95 Proceedings of the 1995 ACM/IEEE conference on Supercomputing
Input/output characteristics of scalable parallel applications
Supercomputing '95 Proceedings of the 1995 ACM/IEEE conference on Supercomputing
Proceedings of the fourth workshop on I/O in parallel and distributed systems: part of the federated computing research conference
Collective parallel I/O
File-Access Characteristics of Parallel Scientific Workloads
IEEE Transactions on Parallel and Distributed Systems
A novel application development environment for large-scale scientific computations
Proceedings of the 14th international conference on Supercomputing
Microprocessor file system interfaces
PDIS '93 Proceedings of the second international conference on Parallel and distributed information systems
Computers and Intractability: A Guide to the Theory of NP-Completeness
Computers and Intractability: A Guide to the Theory of NP-Completeness
Disk Resident Arrays: An Array-Oriented I/O Library for Out-Of-Core Computations
FRONTIERS '96 Proceedings of the 6th Symposium on the Frontiers of Massively Parallel Computation
Design of a Framework for Data-Intensive Wide-Area Applications
HCW '00 Proceedings of the 9th Heterogeneous Computing Workshop
Implementation of algorithms for maximum matching on nonbipartite graphs.
Implementation of algorithms for maximum matching on nonbipartite graphs.
Automatic classification of input/output access patterns
Automatic classification of input/output access patterns
Exploiting Inter-File Access Patterns Using Multi-Collective I/O
FAST '02 Proceedings of the 1st USENIX Conference on File and Storage Technologies
Predicting file system actions from prior events
ATEC '96 Proceedings of the 1996 annual conference on USENIX Annual Technical Conference
Terascale data organization for discovering multivariate climatic trends
Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis
An architecture for a data-intensive computer
Proceedings of the first international workshop on Network-aware data management
Using vector interfaces to deliver millions of IOPS from a networked key-value storage server
Proceedings of the Third ACM Symposium on Cloud Computing
| Hi-index | 0.00 |
The increasing gap between processor cycle times and access times to storage devices makes it necessary to use powerful optimizations. This is especially true for applications in the parallel computing domain that frequently perform large amounts of file I/O. Collective I/O strategy that coordinates the processes to perform I/O on each other's behalf has demonstrated a significant performance improvement. This article proposes a new concept called Multicollective I/O (MCIO) that expands the collective I/O to allow data from multiple files to be requested in a single I/O request, in contrast to allowing only multiple segments for a single file to be specified together. MCIO considers multiple arrays simultaneously by having a more global view of the overall I/O behavior exhibited by parallel applications. This article shows that determining the optimal MCIO access pattern is an NP-complete problem, and proposes two different heuristics for the access pattern detection problem, also called the assignment problem. Both heuristics have been implemented within a runtime library, and tested using a large-scale scientific application. Our results show that MCIO outperforms collective I/O by as much as 87%. Our runtime library-based implementation can be used by application users as well as by optimizing compilers. Based on our results, we recommend that future library designers for I/O-intensive applications include MCIO in their suite of optimizations.