OpenMP: An Industry-Standard API for Shared-Memory Programming
IEEE Computational Science & Engineering
GPFS: A Shared-Disk File System for Large Computing Clusters
FAST '02 Proceedings of the Conference on File and Storage Technologies
A Parallel Algorithm for Extending Cryptographic Hash Functions
INDOCRYPT '01 Proceedings of the Second International Conference on Cryptology in India: Progress in Cryptology
Scalable Unix Commands for Parallel Processors: A High-Performance Implementation
Proceedings of the 8th European PVM/MPI Users' Group Meeting on Recent Advances in Parallel Virtual Machine and Message Passing Interface
MSST '05 Proceedings of the 22nd IEEE / 13th NASA Goddard Conference on Mass Storage Systems and Technologies
The Globus Striped GridFTP Framework and Server
SC '05 Proceedings of the 2005 ACM/IEEE conference on Supercomputing
PVFS: a parallel file system for linux clusters
ALS'00 Proceedings of the 4th annual Linux Showcase & Conference - Volume 4
High speed bulk data transfer using the SSH protocol
Proceedings of the 15th ACM Mardi Gras conference: From lightweight mash-ups to lambda grids: Understanding the spectrum of distributed computing requirements, applications, tools, infrastructures, interoperability, and the incremental adoption of key capabilities
An analysis of data corruption in the storage stack
FAST'08 Proceedings of the 6th USENIX Conference on File and Storage Technologies
IEEE Transactions on Computers
High performance reliable file transfers using automatic many-to-many parallelization
Euro-Par'12 Proceedings of the 18th international conference on Parallel processing workshops
| Hi-index | 0.00 |
Mcp and msum are drop-in replacements for the standard cp and md5sum programs that utilize multiple types of parallelism and other optimizations to achieve maximum copy and checksum performance on clustered file systems. Multi-threading is used to ensure that nodes are kept as busy as possible. Read/write parallelism allows individual operations of a single copy to be overlapped using asynchronous I/O. Multi-node cooperation allows different nodes to take part in the same copy/checksum. Split file processing allows multiple threads to operate concurrently on the same file. Finally, hash trees allow inherently serial checksums to be performed in parallel. This paper presents the design of mcp and msum and detailed performance numbers for each implemented optimization. It will be shown how mcp improves cp performance over 27×, msum improves md5sum performance almost 19×, and the combination of mcp and msum improves verified copies via cp and md5sum by almost 22×.