The grid: blueprint for a new computing infrastructure
The grid: blueprint for a new computing infrastructure
The Globus eXtensible Input/Output System (XIO): A Protocol Independent IO System for the Grid
IPDPS '05 Proceedings of the 19th IEEE International Parallel and Distributed Processing Symposium (IPDPS'05) - Workshop 4 - Volume 05
The Anatomy of the Grid: Enabling Scalable Virtual Organizations
International Journal of High Performance Computing Applications
The Globus Striped GridFTP Framework and Server
SC '05 Proceedings of the 2005 ACM/IEEE conference on Supercomputing
UDT: UDP-based data transfer for high-speed wide area networks
Computer Networks: The International Journal of Computer and Telecommunications Networking
High Performance Data Transfer in Grid Environment Using GridFTP over InfiniBand
CCGRID '10 Proceedings of the 2010 10th IEEE/ACM International Conference on Cluster, Cloud and Grid Computing
Lessons learned from moving earth system grid data sets over a 20 Gbps wide-area network
Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing
A data transfer framework for large-scale science experiments
Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing
A Data Management Framework for Distributed Biomedical Research Environments
E-SCIENCEW '10 Proceedings of the 2010 Sixth IEEE International Conference on e-Science Workshops
IPDPSW '11 Proceedings of the 2011 IEEE International Symposium on Parallel and Distributed Processing Workshops and PhD Forum
Experiences with 100Gbps network applications
Proceedings of the fifth international workshop on Data-Intensive Distributed Computing Date
| Hi-index | 0.00 |
Scientific research creates substantially large volumes of data throughout the processes of discovery and analysis. Given the necessity for data sharing and data relocation, members of the scientific community are often faced with a productivity loss that correlates with the time cost incurred during the data transfer process. The GridFTP protocol was developed to improve this situation by addressing the performance, reliability, and security limitations of standard FTP and other commonly used data movement tools such as SCP. The Globus implementation of GridFTP is widely used to rapidly and reliably move data between geographically distributed systems. Traditionally, GridFTP performs well for datasets containing large files. When the data is partitioned into many small files, however, it suffers from lower transfer rates. Although the pipelining and concurrency solution in GridFTP provides improved transfer rates for datasets using lots-of-small-files, these solutions cannot be applied in environments that have strict firewall rules. In some cases, tarring the files in a dataset on the fly will help; in other cases, a checksum of the files after they are written to disk is desired. In this paper, we present the Globus XIO Pipe Open Driver which enables GridFTP to leverage the standard Unix tools to perform these tasks. We demonstrate the effectiveness of this functionality through several experiments.