The End-to-End Performance Effects of Parallel TCP Sockets on a Lossy Wide-Area Network
IPDPS '02 Proceedings of the 16th International Parallel and Distributed Processing Symposium
Stork: Making Data Placement a First Class Citizen in the Grid
ICDCS '04 Proceedings of the 24th International Conference on Distributed Computing Systems (ICDCS'04)
Modeling and Taming Parallel TCP on the Wide Area Network
IPDPS '05 Proceedings of the 19th IEEE International Parallel and Distributed Processing Symposium (IPDPS'05) - Papers - Volume 01
Adaptive data block scheduling for parallel TCP streams
HPDC '05 Proceedings of the High Performance Distributed Computing, 2005. HPDC-14. Proceedings. 14th IEEE International Symposium
Using overlays for efficient data transfer over shared wide-area networks
Proceedings of the 2008 ACM/IEEE conference on Supercomputing
A new paradigm: Data-aware scheduling in grid computing
Future Generation Computer Systems
Balancing TCP buffer vs parallel streams in application level throughput optimization
Proceedings of the second international workshop on Data-aware distributed computing
A data transfer framework for large-scale science experiments
Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing
Data center networking with multipath TCP
Hotnets-IX Proceedings of the 9th ACM SIGCOMM Workshop on Hot Topics in Networks
A Data Throughput Prediction and Optimization Service for Widely Distributed Many-Task Computing
IEEE Transactions on Parallel and Distributed Systems
Prediction of Optimal Parallelism Level in Wide Area Data Transfers
IEEE Transactions on Parallel and Distributed Systems
Software as a service for data scientists
Communications of the ACM
Network-aware end-to-end data throughput optimization
Proceedings of the first international workshop on Network-aware data management
A Highly-Accurate and Low-Overhead Prediction Model for Transfer Throughput Optimization
SCC '12 Proceedings of the 2012 SC Companion: High Performance Computing, Networking Storage and Analysis
| Hi-index | 0.00 |
Obtaining optimal data transfer performance is of utmost importance to today's data-intensive distributed applications and wide-area data replication services. Doing so necessitates effectively utilizing available network bandwidth and resources, yet in practice transfers seldom reach the levels of utilization they potentially could. Tuning protocol parameters such as pipelining, parallelism, and concurrency can significantly increase utilization and performance, however determining the best settings for these parameters is a difficult problem, as network conditions can vary greatly between sites and over time. In this paper, we present four application-level algorithms for heuristically tuning protocol parameters for data transfers in wide-area networks. Our algorithms dynamically tune the number of parallel data streams per file, the level of control channel pipelining, and the number of concurrent file transfers to fill network pipes. The presented algorithms are implemented as a standalone service as well as being used in interaction with external data scheduling tools such as Stork. The experimental results are very promising, and our algorithms outperform existing solutions in this area.