Massive High-Performance Global File Systems for Grid computing

Authors:
Phil Andrews;Patricia Kovatch;Christopher Jordan
Affiliations:
San Diego Supercomputer Center, La Jolla;San Diego Supercomputer Center, La Jolla;San Diego Supercomputer Center, La Jolla
Venue:
SC '05 Proceedings of the 2005 ACM/IEEE conference on Supercomputing
Year:
2005

Citing 2
Cited 8

Grid Services for Distributed System Integration

Computer
GPFS: A Shared-Disk File System for Large Computing Clusters

FAST '02 Proceedings of the Conference on File and Storage Technologies

A distributed file system for a wide-area high performance computing infrastructure

WORLDS'06 Proceedings of the 3rd conference on USENIX Workshop on Real, Large Distributed Systems - Volume 3
Empowering distributed workflow with the data capacitor: maximizing lustre performance across the wide area network

Proceedings of the 2007 workshop on Service-oriented computing performance: aspects, issues, and approaches
Managing Very-Large Distributed Datasets

OTM '08 Proceedings of the OTM 2008 Confederated International Conferences, CoopIS, DOA, GADA, IS, and ODBASE 2008. Part I on On the Move to Meaningful Internet Systems:
Impact of reservations on production job scheduling

JSSPP'07 Proceedings of the 13th international conference on Job scheduling strategies for parallel processing
A compelling case for a centralized filesystem on the TeraGrid: enhancing an astrophysical workflow with the data capacitor WAN as a test case

Proceedings of the 2010 TeraGrid Conference
Enabling Lustre WAN for production use on the TeraGrid: a lightweight UID mapping scheme

Proceedings of the 2010 TeraGrid Conference
HPC file systems in wide area networks: understanding the performance of lustre over WAN

Euro-Par'12 Proceedings of the 18th international conference on Parallel Processing
A classification of file placement and replication methods on grids

Future Generation Computer Systems

Quantified Score

Hi-index	0.00

Visualization

Abstract

In this paper we describe the evolution of Global File Systems from the concept of a few years ago, to a first demonstration using hardware Fibre Channel frame encoding into IP packets, to a native GFS, to a full prototype demonstration, and finally to a production implementation. The surprisingly excellent performance of the Global File Systems over standard TCP/IP Wide Area Networks has made them a viable candidate for the support of Grid Supercomputing. The implementation designs and performance results are documented within this paper. We also motivate and describe the authentication extensions we made to the IBM GPFS file system, in collaboration with IBM. In several ways Global File Systems are superior to the original approach of wholesale file movement between grid sites and we speculate as to future modes of operation.