Dynamically tuning level of parallelism in wide area data transfers

Authors:
Esma Yildirim;Mehmet Balman;Tevfik Kosar
Affiliations:
Louisiana State University, Baton Rouge, LA, USA;Louisiana State University, Baton Rouge, LA, USA;Louisiana State University, Baton Rouge, LA, USA
Venue:
DADC '08 Proceedings of the 2008 international workshop on Data-aware distributed computing
Year:
2008

Citing 8
Cited 6

Differentiated end-to-end Internet services using a weighted proportional fair sharing TCP

ACM SIGCOMM Computer Communication Review
PSockets: the case for application-level network striping for data intensive applications using high speed wide area networks

Proceedings of the 2000 ACM/IEEE conference on Supercomputing
The End-to-End Performance Effects of Parallel TCP Sockets on a Lossy Wide-Area Network

IPDPS '02 Proceedings of the 16th International Parallel and Distributed Processing Symposium
Stork: Making Data Placement a First Class Citizen in the Grid

ICDCS '04 Proceedings of the 24th International Conference on Distributed Computing Systems (ICDCS'04)
Modeling and Taming Parallel TCP on the Wide Area Network

IPDPS '05 Proceedings of the 19th IEEE International Parallel and Distributed Processing Symposium (IPDPS'05) - Papers - Volume 01
Characterizing and Predicting TCP Throughput on the Wide Area Network

ICDCS '05 Proceedings of the 25th IEEE International Conference on Distributed Computing Systems
Target bandwidth sharing using endhost measures

Performance Evaluation
Adaptive data block scheduling for parallel TCP streams

HPDC '05 Proceedings of the High Performance Distributed Computing, 2005. HPDC-14. Proceedings. 14th IEEE International Symposium

Balancing TCP buffer vs parallel streams in application level throughput optimization

Proceedings of the second international workshop on Data-aware distributed computing
A data transfer framework for large-scale science experiments

Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing
Error detection and error classification: failure awareness in data transfer scheduling

International Journal of Autonomic Computing
Moving huge scientific datasets over the Internet

Concurrency and Computation: Practice & Experience
Experiences with 100Gbps network applications

Proceedings of the fifth international workshop on Data-Intensive Distributed Computing Date
Minimizing the Data Transfer Time Using Multicore End-System Aware Flow Bifurcation

CCGRID '12 Proceedings of the 2012 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (ccgrid 2012)

Quantified Score

Hi-index	0.00

Visualization

Abstract

Using multiple parallel streams for wide area data transfers may yield much better performance than using a single stream, but overwhelming the network by opening too many streams may have an inverse effect. The congestion created by excess number of streams may cause a drop down in the throughput achieved. Hence, it is important to decide on the optimal number of streams without congesting the network. Predicting this 'magic' number is not straightforward, since it depends on many parameters specific to each individual transfer. Generic models that try to predict this number either rely too much on historical information or fail to achieve accurate predictions. In this paper, we present a set of new models which aim to approximate the optimal number with least history information and lowest prediction overhead. We measure the feasibility and accuracy of these models by comparing to actual GridFTP data transfers. We also discuss how these models can be used by a data scheduler to increase the overall performance of the incoming transfer requests.