Reliable and randomized data distribution strategies for large scale storage systems

Authors:
Alberto Miranda;Sascha Effert;Yangwook Kang;Ethan L. Miller;Andre Brinkmann;Toni Cortes
Affiliations:
Barcelona Supercomputing Center (BSC), Barcelona, Spain;University of Paderborn, Paderborn, Germany;University of California, Santa Cruz, CA, USA;University of California, Santa Cruz, CA, USA;University of Paderborn, Paderborn, Germany;Barcelona Supercomputing Center (BSC), Barcelona, Spain
Venue:
HIPC '11 Proceedings of the 2011 18th International Conference on High Performance Computing
Year:
2011

Citing 0
Cited 2

Balls into non-uniform bins

Journal of Parallel and Distributed Computing
CRAID: online RAID upgrades using dynamic hot data reorganization

FAST'14 Proceedings of the 12th USENIX conference on File and Storage Technologies

Quantified Score

Hi-index	0.00

Visualization

Abstract

The ever-growing amount of data requires highly scalable storage solutions. The most flexible approach is to use storage pools that can be expanded and scaled down by adding or removing storage devices. To make this approach usable, it is necessary to provide a solution to locate data items in such a dynamic environment. This paper presents and evaluates the Random Slicing strategy, which incorporates lessons learned from table-based, rule-based, and pseudo-randomized hashing strategies and is able to provide a simple and efficient strategy that scales up to handle exascale data. Random Slicing keeps a small table with information about previous storage system insert and remove operations, drastically reducing the required amount of randomness while delivering a perfect load distribution.