Scalable and Adaptive Metadata Management in Ultra Large-Scale File Systems

Authors:
Yu Hua;Yifeng Zhu;Hong Jiang;Dan Feng;Lei Tian
Affiliations:
-;-;-;-;-
Venue:
ICDCS '08 Proceedings of the 2008 The 28th International Conference on Distributed Computing Systems
Year:
2008

Citing 0
Cited 4

MHS: A distributed metadata management strategy

Journal of Systems and Software
SmartStore: a new metadata organization paradigm with semantic-awareness for next-generation file systems

Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis
Adaptive and scalable metadata management to support a trillion files

Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis
Direct lookup and hash-based metadata placement for local file systems

Proceedings of the 6th International Systems and Storage Conference

Quantified Score

Hi-index	0.00

Visualization

Abstract

This paper presents a scalable and adaptive decentralized metadata lookup scheme for ultra large-scale file systems (≥ Petabytes or even Exabytes). Our scheme logically organizes metadata servers (MDS) into a multi-layered query hierarchy and exploits grouped Bloom filters to efficiently route metadata requests to desired MDS through the hierarchy. This metadata lookup scheme can be executed at the network or memory speed, without being bounded by the performance of slow disks. Our scheme is evaluated through extensive trace-driven simulations and prototype implementation in Linux. Experimental results show that this scheme can significantly improve metadata management scalability and query efficiency in ultra large-scale storage systems.