High-Level Buffering for Hiding Periodic Output Cost in Scientific Simulations
IEEE Transactions on Parallel and Distributed Systems
Constructing collaborative desktop storage caches for large scientific datasets
ACM Transactions on Storage (TOS)
Coupling prefix caching and collective downloads for remote dataset access
Proceedings of the 20th annual international conference on Supercomputing
Semantic-based distributed i/o with the paramedic framework
HPDC '08 Proceedings of the 17th international symposium on High performance distributed computing
Towards a High Performance Implementation of MPI-IO on the Lustre File System
OTM '08 Proceedings of the OTM 2008 Confederated International Conferences, CoopIS, DOA, GADA, IS, and ODBASE 2008. Part I on On the Move to Meaningful Internet Systems:
Implementing a Parallel NetCDF Interface for Seamless Remote I/O Using Multi-dimensional Data
High Performance Computing for Computational Science - VECPAR 2008
Remote MPI-I/O on parallel virtual file system using a circular buffer for high throughput
International Journal of Computers and Applications
MPI-IO/L: efficient remote I/O for MPI-IO via logistical networking
IPDPS'06 Proceedings of the 20th international conference on Parallel and distributed processing
Effective seamless remote MPI-I/O operations with derived data types using PVFS2
EuroPVM/MPI'06 Proceedings of the 13th European PVM/MPI User's Group conference on Recent advances in parallel virtual machine and message passing interface
Remote parallel i/o in grid environments
PPAM'05 Proceedings of the 6th international conference on Parallel Processing and Applied Mathematics
Optimization of nonblocking MPI-I/O to a remote parallel virtual file system using a circular buffer
HPCC'05 Proceedings of the First international conference on High Performance Computing and Communications
Flexible i/o support for reconfigurable grid environments
Euro-Par'06 Proceedings of the 12th international conference on Parallel Processing
| Hi-index | 0.00 |
Scientific applications often need to access remote file systems. Because of slow networks and large data size, however, remote I/O can become an even more serious performance bottleneck than local I/O performance. In this work, we present RFS, a high-performance remote I/O facility for ROMIO, which is a well-known MPI-IO implementation. Our simple, portable, and flexible design eliminates the shortcomings of previous remote I/O efforts. In particular, RFS improves the remote I/O performance by adopting active buffering with threads (ABT), which hides I/O cost by aggressively buffering the output data using available memory and performing background I/O using threads while computation is taking place. Our experimental results show that RFS with ABT can significantly reduce the remote I/O visible cost, achieving up to 92% of the theoretical peak throughput. The computation slowdown caused by concurrent I/O activities was 0.2-6.2%, which is dwarfed by the overall performance improvement in application turnaround time.