A new i/o architecture for improving the performance in large scale clusters

  • Authors:

  • L. M. Sánchez García;Florin D. Isaila;Félix García Carballeira;Jesús Carretero Pérez;Rolf Rabenseifner;Panagiotis Adamidis

  • Affiliations:

  • Computer Science Department, Universidad Carlos III de Madrid, Leganés, Madrid, Spain;Computer Science Department, Universidad Carlos III de Madrid, Leganés, Madrid, Spain;Computer Science Department, Universidad Carlos III de Madrid, Leganés, Madrid, Spain;Computer Science Department, Universidad Carlos III de Madrid, Leganés, Madrid, Spain;High Performance Computing Center Stuttgart (HLRS), Universität Stuttgart, Stuttgart, Germany;High Performance Computing Center Stuttgart (HLRS), Universität Stuttgart, Stuttgart, Germany

  • Venue:
  • ICCSA'06 Proceedings of the 2006 international conference on Computational Science and Its Applications - Volume Part V
  • Year:
  • 2006

Quantified Score

Hi-index 0.03

Visualization

Abstract

The technology advances made in supercomputers and high performance computing clusters over the past few years have been tremendous. Clusters are the most common solution for high performance computing at the present time. In this kind of systems, an important subject is the parallel I/O subsystem design. Parallel file systems (GPFS, PVFS, Lustre, etc) have been the solution used to obtain high performance I/O. Parallel file systems increase performance by distributing data file across several I/O nodes. However, cluster's size is increasing continuously, specially for compute nodes, becoming the I/O nodes in a possible bottleneck of the system. In this paper, we propose a new architecture that solves the problem pointed out before: new hierarchical I/O architecture based on parallel I/O proxies. Those I/O proxies execute on the compute nodes offering an intermediate parallel file system between the applications and the storage system of the cluster. That architecture reduces the load on the I/O nodes increasing the global performance. This paper shows the design of the proposed solution and a preliminary evaluation, using a cluster located in the Stuttgart HLRS center.