A comparative experimental study of parallel file systems for large-scale data processing

Authors:
Zoe Sebepou;Kostas Magoutis;Manolis Marazakis;Angelos Bilas
Affiliations:
Institute of Computer Science, Foundation for Research and Technology - Hellas, Heraklion, Greece and Department of Computer Science, University of Crete, Greece;Institute of Computer Science, Foundation for Research and Technology - Hellas, Heraklion, Greece;Institute of Computer Science, Foundation for Research and Technology - Hellas, Heraklion, Greece;Institute of Computer Science, Foundation for Research and Technology - Hellas, Heraklion, Greece and Department of Computer Science, University of Crete, Greece
Venue:
LASCO'08 First USENIX Workshop on Large-Scale Computing
Year:
2008

Citing 10
Cited 1

Scale and performance in a distributed file system

ACM Transactions on Computer Systems (TOCS)
A cost-effective, high-bandwidth storage architecture

Proceedings of the eighth international conference on Architectural support for programming languages and operating systems
MPI-The Complete Reference, Volume 1: The MPI Core

MPI-The Complete Reference, Volume 1: The MPI Core
GPFS: A Shared-Disk File System for Large Computing Clusters

FAST '02 Proceedings of the Conference on File and Storage Technologies
Structure and Performance of the Direct Access File System

ATEC '02 Proceedings of the General Track of the annual conference on USENIX Annual Technical Conference
Noncontiguous I/O through PVFS

CLUSTER '02 Proceedings of the IEEE International Conference on Cluster Computing
The Case for Application-Specific Benchmarking

HOTOS '99 Proceedings of the The Seventh Workshop on Hot Topics in Operating Systems
Workload-specific file system benchmarks

Workload-specific file system benchmarks
Performance Evaluation of the PVFS2 Architecture

PDP '07 Proceedings of the 15th Euromicro International Conference on Parallel, Distributed and Network-Based Processing
Scalable performance of the Panasas parallel file system

FAST'08 Proceedings of the 6th USENIX Conference on File and Storage Technologies

Performance optimization of small file i/o with adaptive migration strategy in cluster file system

HPCA'09 Proceedings of the Second international conference on High Performance Computing and Applications

Quantified Score

Hi-index	0.00

Visualization

Abstract

Large-scale scientific and business applications require data processing of ever-increasing amounts of data, fueling a demand for scalable parallel file systems comprising hundreds to thousands of disks. Modern parallel file system architectures however, span a large and complex design space. As a result, IT architects are faced with a challenge when deciding on the most appropriate parallel file system for a specific scientific or industrial application in a large-scale computing installation. Typically, the right choice depends on the characteristics of the application as well as the design assumptions built into a parallel file system. In this study, we take a close look at two prominent modern parallel file systems, PVFS2 and Lustre, and compare them experimentally on a range of benchmark-driven scenarios modeling specific real-world applications.