A framework for self-optimizing, fault-tolerant, high performance bulk data transfers in a heterogeneous grid environment

Authors:
Tevfik Kosar;George Kola;Miron Livny
Affiliations:
Computer Sciences Department, University of Wisconsin-Madison, Madison WI;Computer Sciences Department, University of Wisconsin-Madison, Madison WI;Computer Sciences Department, University of Wisconsin-Madison, Madison WI
Venue:
ISPDC'03 Proceedings of the Second international conference on Parallel and distributed computing
Year:
2003

Citing 6
Cited 1

DEVise: integrated querying and visual exploration of large datasets

SIGMOD '97 Proceedings of the 1997 ACM SIGMOD international conference on Management of data
The Globus toolkit

The grid
The SDSC storage resource broker

CASCON '98 Proceedings of the 1998 conference of the Centre for Advanced Studies on Collaborative research
Secure, Efficient Data Transport and Replica Management for High-Performance Data-Intensive Computing

MSS '01 Proceedings of the Eighteenth IEEE Symposium on Mass Storage Systems and Technologies
Condor-G: A Computation Management Agent for Multi-Institutional Grids

HPDC '01 Proceedings of the 10th IEEE International Symposium on High Performance Distributed Computing
The Kangaroo Approach to Data Movement on the Grid

HPDC '01 Proceedings of the 10th IEEE International Symposium on High Performance Distributed Computing

On multipath routing with transit hubs

NETWORKING'05 Proceedings of the 4th IFIP-TC6 international conference on Networking Technologies, Services, and Protocols; Performance of Computer and Communication Networks; Mobile and Wireless Communication Systems

Quantified Score

Hi-index	0.00

Visualization

Abstract

The drastic increase in the data requirements of scientific applications combined with an increasing trend towards collaborative research has resulted in the need to transfer large amounts of data among the participating sites. The general approach to transferring such large amounts of data has been to either dump data to tapes and mail them or employ scripts with an operator at each site to babysit the transfers to deal with failures. We introduce a framework which automates the whole process of data movement between different sites. The framework does not require any human intervention and it can recover automatically from various kinds of storage system, network, and software failures, guaranteeing completion of the transfers. The framework has sophisticated monitoring and tuning capability that increases the performance of the data transfers on the fly. The framework also generates on-the-fly visualization of the transfers making identification of problems and bottlenecks in the system simple.