The End-to-End Performance Effects of Parallel TCP Sockets on a Lossy Wide-Area Network
IPDPS '02 Proceedings of the 16th International Parallel and Distributed Processing Symposium
Bulk data transfer forecasts and the implications to grid scheduling
Bulk data transfer forecasts and the implications to grid scheduling
Stork: Making Data Placement a First Class Citizen in the Grid
ICDCS '04 Proceedings of the 24th International Conference on Distributed Computing Systems (ICDCS'04)
Modeling and Taming Parallel TCP on the Wide Area Network
IPDPS '05 Proceedings of the 19th IEEE International Parallel and Distributed Processing Symposium (IPDPS'05) - Papers - Volume 01
The Globus Striped GridFTP Framework and Server
SC '05 Proceedings of the 2005 ACM/IEEE conference on Supercomputing
Dynamically tuning level of parallelism in wide area data transfers
DADC '08 Proceedings of the 2008 international workshop on Data-aware distributed computing
A new paradigm: Data-aware scheduling in grid computing
Future Generation Computer Systems
Secure, Performance-Oriented Data Management for nanoCMOS Electronics
ESCIENCE '08 Proceedings of the 2008 Fourth IEEE International Conference on eScience
Globus XIO pipe open driver: enabling GridFTP to leverage standard Unix tools
Proceedings of the 2011 TeraGrid Conference: Extreme Digital Discovery
Network-aware end-to-end data throughput optimization
Proceedings of the first international workshop on Network-aware data management
End-to-End Data-Flow Parallelism for Throughput Optimization in High-Speed Networks
Journal of Grid Computing
ATLAS grid workload on NDGF resources: analysis, modeling, and workload generation
SC '12 Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis
Taming massive distributed datasets: data sampling using bitmap indices
Proceedings of the 22nd international symposium on High-performance parallel and distributed computing
StorkCloud: data transfer scheduling and optimization as a service
Proceedings of the 4th ACM workshop on Scientific cloud computing
Modeling throughput sampling size for a cloud-hosted data scheduling and optimization service
Future Generation Computer Systems
SDQuery DSI: integrating data management support with a wide area data transfer protocol
SC '13 Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis
Dynamic protocol tuning algorithms for high performance data transfers
Euro-Par'13 Proceedings of the 19th international conference on Parallel Processing
| Hi-index | 0.00 |
Modern scientific experiments can generate hundreds of gigabytes to terabytes or even petabytes of data that may furthermore be maintained in large numbers of relatively small files. Frequently, this data must be disseminated to remote collaborators or computational centers for data analysis. Moving this data with high performance and strong robustness and providing a simple interface for users are challenging tasks. We present a data transfer framework comprising a high-performance data transfer library based on GridFTP, a data scheduler, and a graphical user interface that allows users to transfer their data easily, reliably, and securely. This system incorporates automatic tuning mechanisms to select at runtime the number of concurrent threads to be used for transfers. Also included are restart mechanisms capable of dealing with client, network, and server failures. Experimental results indicate that our data transfer system can significantly improve data transfer performance and can recover well from failures.