Analyzing Metadata Performance in Distributed File Systems

  • Authors:
  • Christoph Biardzki;Thomas Ludwig

  • Affiliations:
  • Leibniz-Rechenzentrum (LRZ) der Bayerischen Akademie der Wissenschaften, Garching, Germany 85748;German Climate Computing Centre (DKRZ), Hamburg, Germany 20146

  • Venue:
  • PaCT '09 Proceedings of the 10th International Conference on Parallel Computing Technologies
  • Year:
  • 2009

Quantified Score

Hi-index 0.00

Visualization

Abstract

The performance of metadata processing in large distributed file systems currently presents larger challenges than scaling of data throughput. The paper presents a novel, distributed benchmark called DMetabench for measuring the performance of metadata operations (e.g. file creation). DMetabench runs in environments with potentially thousands of nodes and allows an assessment of the scalability of metadata operations. Additionally, precise run-time performance data is preserved which allows for a better understanding of performance artifacts. Validation results from production file systems at the Leibniz Supercomputing Centre (LRZ) are provided and discussed. Possible applications of knowledge about metadata performance scaling include the choice of an optimal parallelization strategy for metadata-itensive workload in a specific runtime environment.