HPC file systems in wide area networks: understanding the performance of lustre over WAN

Authors:
Alvaro Aguilera;Michael Kluge;Thomas William;Wolfgang E. Nagel
Affiliations:
Technische Universität Dresden, Dresden, Germany;Technische Universität Dresden, Dresden, Germany;Technische Universität Dresden, Dresden, Germany;Technische Universität Dresden, Dresden, Germany
Venue:
Euro-Par'12 Proceedings of the 18th international conference on Parallel Processing
Year:
2012

Citing 6
Cited 0

The performance of TCP/IP for networks with high bandwidth-delay products and random loss

IEEE/ACM Transactions on Networking (TON)
Massive High-Performance Global File Systems for Grid computing

SC '05 Proceedings of the 2005 ACM/IEEE conference on Supercomputing
All in a day's work: advancing data-intensive research with the data capacitor

Proceedings of the 2006 ACM/IEEE conference on Supercomputing
Empowering distributed workflow with the data capacitor: maximizing lustre performance across the wide area network

Proceedings of the 2007 workshop on Service-oriented computing performance: aspects, issues, and approaches
A compelling case for a centralized filesystem on the TeraGrid: enhancing an astrophysical workflow with the data capacitor WAN as a test case

Proceedings of the 2010 TeraGrid Conference
Evaluation of a Performance Model of Lustre File System

CHINAGRID '10 Proceedings of the The Fifth Annual ChinaGrid Conference

Quantified Score

Hi-index	0.00

Visualization

Abstract

Using the first commercially available 100 Gbps Ethernet technology with a link of varying length, we have evaluated the performance of the Lustre file system and its networking layer under different latency scenarios. The results led us to a better understanding of the impact that the network latency has on Lustre's performance. In particular spanning Lustre's networking layer, striped small I/O, and the parallel creation of files inside a common directory. The main contribution of this work is the derivation of useful rules of thumbs to help users and system administrators predict the variation in Lustre's performance produced as a result of changes in the latency of the I/O network.