High-Bandwidth remote parallel i/o with the distributed memory filesystem MEMFS

Authors:
Jan Seidel;Rudolf Berrendorf;Marcel Birkner;Marc-André Hermanns
Affiliations:
Department of Computer Science, University of Applied Sciences Bonn-Rhein-Sieg, St. Augustin, Germany;Department of Computer Science, University of Applied Sciences Bonn-Rhein-Sieg, St. Augustin, Germany;Department of Computer Science, University of Applied Sciences Bonn-Rhein-Sieg, St. Augustin, Germany;Central Institute for Applied Mathematics, Research Centre Jülich, Jülich, Germany
Venue:
EuroPVM/MPI'06 Proceedings of the 13th European PVM/MPI User's Group conference on Recent advances in parallel virtual machine and message passing interface
Year:
2006

Citing 5
Cited 1

The Network RamDisk: Using remote memory on heterogeneous NOWs

Cluster Computing
An Abstract-Device Interface for Implementing Portable Parallel-I/O Interfaces

FRONTIERS '96 Proceedings of the 6th Symposium on the Frontiers of Massively Parallel Computation
A Message Passing Interface Library for Inhomogeneous Coupled Clusters

IPDPS '03 Proceedings of the 17th International Symposium on Parallel and Distributed Processing
The Google file system

SOSP '03 Proceedings of the nineteenth ACM symposium on Operating systems principles
Remote parallel i/o in grid environments

PPAM'05 Proceedings of the 6th international conference on Parallel Processing and Applied Mathematics

A 1 PB/s file system to checkpoint three million MPI tasks

Proceedings of the 22nd international symposium on High-performance parallel and distributed computing

Quantified Score

Hi-index	0.00

Visualization

Abstract

The enormous advance in computational power of supercomputers enables scientific applications to process problems of increasing size. This is often correlated with an increasing amount of data stored in (parallel) filesystems. As the increase in bandwith of common disk based I/O devices can not keep up with the evolution of computational power, the access to this data becomes the bottleneck in many applications. memfs takes the approach to distribute I/O data among multiple dedicated remote servers on a user-level basis. It stores files in the accumulated main memory of these I/O nodes and is able to deliver this data with high bandwidth. We describe how memfs manages a memory based distributed filesystem, how it stores data among the participating I/O servers and how it assigns servers to application clients. Results are given for a usage in a grid project with high-bandwidth wan connections.