`Perfectly' Scalable Data I/O

  • Authors:
  • Steven A. Orszag;Isaac Goldhirsch;Sudhir Srinivasan

  • Affiliations:
  • Department of Mathematics, Yale University, New Haven 06520-8283;Department of Fluid Mechanics and Heat Transfer, Tel-Aviv University, Tel-Aviv, Israel 69978;Ibrix, Inc., Billerica 01854

  • Venue:
  • Journal of Scientific Computing
  • Year:
  • 2005

Quantified Score

Hi-index 0.01

Visualization

Abstract

This paper describes and analyzes a new architecture for file systems in which `metadata', lock control, etc., are distributed among diverse resources. The basic data structure is a segment, viz. a logical group of files, folders, or other objects. The file system requires only one root, and can be non-hierarchical without a complete tree structure within segments. For `embarrassingly parallel' data distributions, scalability is trivially perfect for all N,where N is the number of servers. Even for random file access, a new extreme statistical mechanics is used to show that data I/O is `perfectly' scalable with probability 1, with degradation from perfect scaling that is small and bounded by f ln N/ ln (ln N). Here f is the fraction of data that is metadata. In contrast, earlier solutions degrade much faster, like Nf. No structural changes in classical metadata are required.