GIGA+: scalable directories for shared file systems

Authors:
Swapnil V. Patil;Garth A. Gibson;Sam Lang;Milo Polte
Affiliations:
Carnegie Mellon University;Carnegie Mellon University;Argonne National Lab;Carnegie Mellon University
Venue:
PDSW '07 Proceedings of the 2nd international workshop on Petascale data storage: held in conjunction with Supercomputing '07
Year:
2007

Citing 9
Cited 6

Availability in the Sprite distributed file system

ACM SIGOPS Operating Systems Review
Efficient locking for concurrent operations on B-trees

ACM Transactions on Database Systems (TODS)
Extendible hashing—a fast access method for dynamic files

ACM Transactions on Database Systems (TODS)
A trace-driven analysis of the UNIX 4.2 BSD file system

Proceedings of the tenth ACM symposium on Operating systems principles
GPFS: A Shared-Disk File System for Large Computing Clusters

FAST '02 Proceedings of the Conference on File and Storage Technologies
Planned Extensions to the Linux Ext2/Ext3 Filesystem

Proceedings of the FREENIX Track: 2002 USENIX Annual Technical Conference
Boxwood: abstractions as the foundation for storage infrastructure

OSDI'04 Proceedings of the 6th conference on Symposium on Opearting Systems Design & Implementation - Volume 6
Data ONTAP GX: a scalable storage cluster

FAST '07 Proceedings of the 5th USENIX conference on File and Storage Technologies
Scalability in the XFS file system

ATEC '96 Proceedings of the 1996 annual conference on USENIX Annual Technical Conference

PLFS: a checkpoint filesystem for parallel applications

Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis
Adaptive and scalable metadata management to support a trillion files

Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis
Finding a needle in Haystack: facebook's photo storage

OSDI'10 Proceedings of the 9th USENIX conference on Operating systems design and implementation
Parallel I/O and the metadata wall

Proceedings of the sixth workshop on Parallel Data Storage
An approach for indexing file names in a directory

Proceedings of the 13th International Conference on Computer Systems and Technologies
MTC envelope: defining the capability of large scale computers in the context of parallel scripting applications

Proceedings of the 22nd international symposium on High-performance parallel and distributed computing

Quantified Score

Hi-index	0.00

Visualization

Abstract

There is an increasing use of high-performance computing (HPC) clusters with thousands of compute nodes that, with the advent of multi-core CPUs, will impose a significant challenge for storage systems: The ability to scale to handle I/O generated by applications executing in parallel in tens of thousands of threads. One such challenge is building scalable directories for cluster storage - i.e., directories that can store billions to trillions of entries and handle hundreds of thousands of operations per second.