Architectural requirements of parallel scientific applications with explicit communication
ISCA '93 Proceedings of the 20th annual international symposium on computer architecture
The high performance storage system
Proceedings of the 1993 ACM/IEEE conference on Supercomputing
Extensible file system (ELFS): an object-oriented approach to high performance file I/O
OOPSLA '94 Proceedings of the ninth annual conference on Object-oriented programming systems, language, and applications
Parallel file systems for the IBM SP computers
IBM Systems Journal
Server-directed collective I/O in Panda
Supercomputing '95 Proceedings of the 1995 ACM/IEEE conference on Supercomputing
Input/output characteristics of scalable parallel applications
Supercomputing '95 Proceedings of the 1995 ACM/IEEE conference on Supercomputing
File-Access Characteristics of Parallel Scientific Workloads
IEEE Transactions on Parallel and Distributed Systems
A novel application development environment for large-scale scientific computations
Proceedings of the 14th international conference on Supercomputing
Design of a Framework for Data-Intensive Wide-Area Applications
HCW '00 Proceedings of the 9th Heterogeneous Computing Workshop
Efficient I/O for parallel visualization
EG PGV'11 Proceedings of the 11th Eurographics conference on Parallel Graphics and Visualization
| Hi-index | 0.00 |
This paper introduces a new concept called Multi-Collective I/O (MCIO) that extends conventional collective I/O to optimize I/O accesses to multiple arrays simultaneously. In this approach, as in collective I/O, multiple processors co-ordinate to perform I/O on behalf of each other if doing so improves overall I/O time. However, unlike collective I/O, MCIO considers multiple arrays simultaneously; that is, it has a more global view of the overall I/O behavior exhibited by application. This paper shows that determining optimal MCIO access pattern is an NP-complete problem, and proposes two different heuristics for the access pattern detection problem (also called the assignment problem). Both of the heuristics have been implemented within a runtime library, and tested using a large-scale scientific application. Our preliminary results show that MCIO outperforms collective I/O by as much as 87%. Our runtime library-based implementation can be used by users as well as optimizing compilers. Based on our results, we recommend future library designers for I/O-intensive applications to include MCIO in their suite of optimizations.