Performance optimization of small file i/o with adaptive migration strategy in cluster file system

Authors:
Xiuqiao Li;Bin Dong;Limin Xiao;Li Ruan
Affiliations:
School of Computer Science and Engineering, BeiHang University, Beijing, P.R. China;School of Computer Science and Engineering, BeiHang University, Beijing, P.R. China;School of Computer Science and Engineering, BeiHang University, Beijing, P.R. China;School of Computer Science and Engineering, BeiHang University, Beijing, P.R. China
Venue:
HPCA'09 Proceedings of the Second international conference on High Performance Computing and Applications
Year:
2009

Citing 4
Cited 0

Large files, small writes, and pNFS

Proceedings of the 20th annual international conference on Supercomputing
A comparison of file system workloads

ATEC '00 Proceedings of the annual conference on USENIX Annual Technical Conference
A comparative experimental study of parallel file systems for large-scale data processing

LASCO'08 First USENIX Workshop on Large-Scale Computing
Directory-Based Metadata Optimizations for Small Files in PVFS

Euro-Par '08 Proceedings of the 14th international Euro-Par conference on Parallel Processing

Quantified Score

Hi-index	0.00

Visualization

Abstract

While cluster file systems exploit data striping scheme to boost large file I/O throughput, small file performance is impaired and neglected. Common metadata-based optimizations introduce obstacles such as metadata server overload and migration latency. In this paper, a novel adaptive migration strategy is incorporated into metadata-based optimization to alleviate these side effects by migrating file dynamically. Guided by proposed adaptive migration threshold model, two types of file migration are applied to reduce metadata server load without degrading current performance of file system obviously. Schemes of latency hiding and migration consistency are also introduced to reduce overhead induced by small file optimization. Our results indicate that proposed optimization can substantially improve file creation and deletion performance, and boost small file I/O throughput by more than 20%. Moreover, side effects on overall performance produced by file migration are slight and can be absorbed by improvements.