Managing very large distributed data sets on a data grid

Authors:
Miguel Branco;Ed Zaluska;David de Roure;Mario Lassnig;Vincent Garonne
Affiliations:
CERN, European Organization for Nuclear Research, Switzerland and University of Southampton, U.K.;University of Southampton, U.K.;University of Southampton, U.K.;CERN, European Organization for Nuclear Research, Switzerland and University of Innsbruck, Austria;CERN, European Organization for Nuclear Research, Switzerland
Venue:
Concurrency and Computation: Practice & Experience - Grid Computing, High Performance and Distributed Application
Year:
2010

Citing 0
Cited 2

Identification, Modelling and Prediction of Non-periodic Bursts in Workloads

CCGRID '10 Proceedings of the 2010 10th IEEE/ACM International Conference on Cluster, Cloud and Grid Computing
A similarity measure for time, frequency, and dependencies in large-scale workloads

Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis

Quantified Score

Hi-index	0.00

Visualization

Abstract

In this work we address the management of very large data sets, which need to be stored and processed across many computing sites. The motivation for our work is the ATLAS experiment for the Large Hadron Collider (LHC), where the authors have been involved in the development of the data management middleware. This middleware, called DQ2, has been used for the last several years by the ATLAS experiment for shipping petabytes of data to research centres and universities worldwide. We describe our experience in developing and deploying DQ2 on the Worldwide LHC computing Grid, a production Grid infrastructure formed of hundreds of computing sites. From this operational experience, we have identified an important degree of uncertainty that underlies the behaviour of large Grid infrastructures. This uncertainty is subjected to a detailed analysis, leading us to present novel modelling and simulation techniques for Data Grids. In addition, we discuss what we perceive as practical limits to the development of data distribution algorithms for Data Grids given the underlying infrastructure uncertainty, and propose future research directions. Copyright © 2009 John Wiley & Sons, Ltd.