Optimization of nonblocking MPI-I/O to a remote parallel virtual file system using a circular buffer

  • Authors:

  • Yuichi Tsujita

  • Affiliations:

  • Department of Electronic Engineering and Computer Science, Faculty of Engineering, Kinki University, Hiroshima, Japan

  • Venue:
  • HPCC'05 Proceedings of the First international conference on High Performance Computing and Communications
  • Year:
  • 2005

Quantified Score

Hi-index 0.00

Visualization

Abstract

Parallel computation applications output intermediate data periodically, and typically the outputs are moved to a remote computer for visualization. A flexible intermediate library named Stampi realizes seamless MPI-I/O operations both inside a computer and among computers. MPI-I/O operations to a remote computer are realized by its MPI-I/O processes which are invoked on a remote computer. To realize data-intensive I/O operations, a Parallel Virtual File System (PVFS) was supported in the MPI-I/O mechanism. MPI-I/O operations to a PVFS file system on a remote computer are available with seamless interfaces of the Stampi library. Among many kinds of I/O functions, nonblocking MPI-I/O functions provide overlap of computation with I/O operations, and visible I/O times can be minimized with them. Due to its architectural constraints and slow network, visible I/O times of them became long with an increase in the number of user processes and message data size. To minimize the times, a circular buffer system has been implemented in the mechanism. With the help of the circular buffer, the visible I/O times have been minimized effectively.