The design and implementation of a log-structured file system
ACM Transactions on Computer Systems (TOCS)
Reliable File Transfer in Grid Environments
LCN '02 Proceedings of the 27th Annual IEEE Conference on Local Computer Networks
The Globus Striped GridFTP Framework and Server
SC '05 Proceedings of the 2005 ACM/IEEE conference on Supercomputing
A framework for reliable and efficient data placement in distributed computing systems
Journal of Parallel and Distributed Computing - Special issue: Design and performance of networks for super-, cluster-, and grid-computing: Part I
Mesh: secure, lightweight grid middleware using existing SSH infrastructure
Proceedings of the 12th ACM symposium on Access control models and technologies
High speed bulk data transfer using the SSH protocol
Proceedings of the 15th ACM Mardi Gras conference: From lightweight mash-ups to lambda grids: Understanding the spectrum of distributed computing requirements, applications, tools, infrastructures, interoperability, and the incremental adoption of key capabilities
Replica Aware Reliable File Transfer Service for the Data Grid
ESCIENCE '08 Proceedings of the 2008 Fourth IEEE International Conference on eScience
Future Generation Computer Systems
High performance multi-node file copies and checksums for clustered file systems
LISA'10 Proceedings of the 24th international conference on Large installation system administration
| Hi-index | 0.00 |
Shift is a lightweight framework for high performance local and remote file transfers that provides resiliency across a wide variety of failure scenarios. Shift supports multiple file transport protocols with automatic selection of the most appropriate mechanism between each pair of participating hosts allowing it to adapt to heterogeneous clients with differing software and network access restrictions. File system information is gathered from clients and servers to detect file system equivalence and enable path rewriting so that multiple clients can be automatically spawned in parallel to carry out both single and multi-file transfers to multiple servers selected according to load and availability. This improves both reliability and performance by eliminating single points of failure and overcoming single system bottlenecks. End-to-end integrity is provided using cryptographic hashes at the source and destination with support for partial file retransmission of only corrupted portions. This paper presents the design and implementation of Shift and details the mechanisms utilized to enhance the reliability and performance of file transfers.