System Performance Tuning
Data management and transfer in high-performance computational grid environments
Parallel Computing - Parallel data-intensive algorithms and applications
SnapMirror: File-System-Based Asynchronous Mirroring for Disaster Recovery
FAST '02 Proceedings of the Conference on File and Storage Technologies
Data Management in an International Data Grid Project
GRID '00 Proceedings of the First IEEE/ACM International Workshop on Grid Computing
The SDSC storage resource broker
CASCON '98 Proceedings of the 1998 conference of the Centre for Advanced Studies on Collaborative research
File and Object Replication in Data Grids
HPDC '01 Proceedings of the 10th IEEE International Symposium on High Performance Distributed Computing
The Kangaroo Approach to Data Movement on the Grid
HPDC '01 Proceedings of the 10th IEEE International Symposium on High Performance Distributed Computing
Design and implementation of an enterprise grid
IBM Systems Journal
Evaluation of service selection techniques in service oriented computing networks
Multiagent and Grid Systems - Smart Grid Technologies & Market Models
Toward Seamless Grid Data Access: Design and Implementation of GridFTP on .NET
GRID '05 Proceedings of the 6th IEEE/ACM International Workshop on Grid Computing
Efficient access to many samall files in a filesystem for grid computing
GRID '07 Proceedings of the 8th IEEE/ACM International Conference on Grid Computing
| Hi-index | 0.00 |
The Grid communities efforts on managing and transporting data have focused on very large data sets consisting of very large elements. We are interested in leveraging the benefits of solutions such GridFTP, in particular with respect to parallel data transfer and restartability (as well as security, third party control, etc.), for moving large data sets consisting of very large numbers of small objects, e.g., moving a file system subtree. In addition, we require a solution that 1) imposes constant memory overhead on the client and server systems, 2) is independent of the actual transfer mechanism used so we can easily take advantage of advances in technologies for transferring large files, 3) works well even for very large collections of very small files and 4) is a complete solution, i.e., reproduces the directory tree at the server. In this paper, we present DYNAMO which is our tool built on GridFTP for transferring directory subtrees. In addition to describing the architecture and implementation of DYNAMO, we present performance results on a range of networks showing that we have met our goals.