MagPIe: MPI's collective communication operations for clustered wide area systems
Proceedings of the seventh ACM SIGPLAN symposium on Principles and practice of parallel programming
Building a high-performance collective communication library
Proceedings of the 1994 ACM/IEEE conference on Supercomputing
SOSP '03 Proceedings of the nineteenth ACM symposium on Operating systems principles
SplitStream: high-bandwidth multicast in cooperative environments
SOSP '03 Proceedings of the nineteenth ACM symposium on Operating systems principles
Ibis: a flexible and efficient Java-based Grid programming environment: Research Articles
Concurrency and Computation: Practice & Experience - 2002 ACM Java Grande–ISCOPE Conference Part II
Balanced Multicasting: High-throughput Communication for Grid Applications
SC '05 Proceedings of the 2005 ACM/IEEE conference on Supercomputing
High-Performance MPI Broadcast Algorithm for Grid Environments Utilizing Multi-lane NICs
CCGRID '07 Proceedings of the Seventh IEEE International Symposium on Cluster Computing and the Grid
A fast topology inference: a building block for network-aware parallel processing
Proceedings of the 16th international symposium on High performance distributed computing
MOB: zero-configuration high-throughput multicasting for grid applications
Proceedings of the 16th international symposium on High performance distributed computing
CCGRID '08 Proceedings of the 2008 Eighth IEEE International Symposium on Cluster Computing and the Grid
Amazon S3 for science grids: a viable solution?
DADC '08 Proceedings of the 2008 international workshop on Data-aware distributed computing
The cost of doing science on the cloud: the Montage example
Proceedings of the 2008 ACM/IEEE conference on Supercomputing
Can cloud computing reach the top500?
Proceedings of the combined workshops on UnConventional high performance computing workshop plus memory access workshop
Collective Receiver-Initiated Multicast for Grid Applications
IEEE Transactions on Parallel and Distributed Systems
Bayesian Cognitive Model in Scheduling Algorithm for Data Intensive Computing
Journal of Grid Computing
Journal of Computer and System Sciences
International Journal of Parallel Programming
| Hi-index | 0.00 |
Data-intensive parallel applications on clouds need to deploy large data sets from the cloud's storage facility to all compute nodes as fast as possible. Many multicast algorithms have been proposed for clusters and grid environments. The most common approach is to construct one or more spanning trees based on the network topology and network monitoring data in order to maximize available bandwidth and avoid bottleneck links. However, delivering optimal performance becomes difficult once the available bandwidth changes dynamically. In this paper, we focus on Amazon EC2/S3 (the most commonly used cloud platform today) and propose two high performance multicast algorithms. These algorithms make it possible to efficiently transfer large amounts of data stored in Amazon S3 to multiple Amazon EC2 nodes. The three salient features of our algorithms are (1) to construct an overlay network on clouds without network topology information, (2) to optimize the total throughput dynamically, and (3) to increase the download throughput by letting nodes cooperate with each other. The two algorithms differ in the way nodes cooperate: the first `non-steal' algorithm lets each node download an equal share of all data, while the second `steal' algorithm uses work stealing to counter the effect of heterogeneous download bandwidth. As a result, all nodes can download files from S3 quickly, even when the network performance changes while the algorithm is running. We evaluate our algorithms on EC2/S3, and show that they are scalable and consistently achieve high throughput. Both algorithms perform much better than having each node downloading all data directly from S3.