Empirical Analysis of a Large-Scale Hierarchical Storage System
Euro-Par '08 Proceedings of the 14th international Euro-Par conference on Parallel Processing
Latency Hiding File I/O for Blue Gene Systems
CCGRID '09 Proceedings of the 2009 9th IEEE/ACM International Symposium on Cluster Computing and the Grid
Multiple-Level MPI File Write-Back and Prefetching for Blue Gene Systems
Proceedings of the 16th European PVM/MPI Users' Group Meeting on Recent Advances in Parallel Virtual Machine and Message Passing Interface
Managing Variability in the IO Performance of Petascale Storage Systems
Proceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis
| Hi-index | 0.00 |
Parallel I/O over Cray XT is supported by a vendor-supplied MPI-IO package. This package contains a proprietary ADIO implementation built on top of the sysio library. While it is reasonable to maintain a stable code base for users' convenience, it is also very important for the system developers and researchers to analyze and assess the effectiveness of parallel I/O software, and accordingly, tune and optimize the MPI-IO implementation. A proprietary parallel I/O code base relinquishes such flexibilities. On the other hand, a generic UFS-based MPI-IO implementation is typically used on many Linux-based platforms, which lacks of the capability of utilizing Lustre specific features. We have developed an opensource MPI-IO package over Lustre, referred to as OPAL (OPportunistic and Adaptive MPI-IO Library over Lustre). OPAL provides a single source-code base for MPI-IO over Lustre on both the Cray XT and Linux platforms. It provides a number of good features, including arbitrary specification of striping patterns and Lustre-stripe aligned file domain partitioning. This paper presents the performance comparisons between OPAL and Cray's proprietary implementation. Our evaluation demonstrates that OPAL achieves the performance comparable to the Cray implementation. We also exemplify the benefits of OPAL in profiling the internal time breakdown of the collective I/O protocol over Cray XT.