Improving GridFTP transfers by means of a multiagent parallel file system
Multiagent and Grid Systems - Grid Computing, high performance and distributed applications
DIMM: a distributed metadata management for data-intensive HPC environments
DADC '08 Proceedings of the 2008 international workshop on Data-aware distributed computing
GRIMS: a scalable management and storage system for massive remote sensing images
Proceedings of the 3rd international conference on Scalable information systems
Security requirements analysis for large-scale distributed file systems
Euro-Par'06 Proceedings of the CoreGRID 2006, UNICORE Summit 2006, Petascale Computational Biology and Bioinformatics conference on Parallel processing
FDTM: block level data migration policy in tiered storage system
NPC'10 Proceedings of the 2010 IFIP international conference on Network and parallel computing
A parallel data storage interface to GridFTP
ODBASE'06/OTM'06 Proceedings of the 2006 Confederated international conference on On the Move to Meaningful Internet Systems: CoopIS, DOA, GADA, and ODBASE - Volume Part II
| Hi-index | 0.00 |
The High Performance Storage System (HPSS) provides scalable hierarchical storage management (HSM), archive, and file system services. Its design, implementation and current dominant use are focused on HSM and archive services. It is also a general-purpose, global, shared, parallel file system, potentially useful in other application domains. When HPSS design and implementation began over a decade ago, scientific computing power and storage capabilities at a site, such as a DOE national laboratory, was measured in a few 10s of gigaops, data archived in HSMs in a few 10s of terabytes at most, data throughput rates to an HSM in a few megabytes/s, and daily throughput with the HSM in a few gigabytes/day. At that time, the DOE national laboratories and IBM HPSS design team recognized that we were headed for a data storage explosion driven by computing power rising to teraops/petaops requiring data stored in HSMs to rise to petabytes and beyond, data transfer rates with the HSM to rise to gigabytes/s and higher, and daily throughput with a HSM in 10s of terabytes/day. This paper discusses HPSS architectural, implementation and deployment experiences that contributed to its success in meeting the above orders of magnitude scaling targets. We also discuss areas that need additional attention as we continue significant scaling into the future.