Data management and transfer in high-performance computational grid environments

Authors:
Bill Allcock;Joe Bester;John Bresnahan;Ann L. Chervenak;Ian Foster;Carl Kesselman;Sam Meder;Veronika Nefedova;Darcy Quesnel;Steven Tuecke
Affiliations:
Mathematics and Computer Science Division, Argonne National Laboratory, Argonne, IL;Mathematics and Computer Science Division, Argonne National Laboratory, Argonne, IL;Mathematics and Computer Science Division, Argonne National Laboratory, Argonne, IL;Information Sciences Institute, University of Southern California, Los Angeles, CA;Mathematics and Computer Science Division, Argonne National Laboratory, Argonne, IL and Department of Computer Science, The Computation Institute, The University of Chicago, Chicago, IL;Information Sciences Institute, University of Southern California, Los Angeles, CA;Mathematics and Computer Science Division, Argonne National Laboratory, Argonne, IL;Mathematics and Computer Science Division, Argonne National Laboratory, Argonne, IL;Mathematics and Computer Science Division, Argonne National Laboratory, Argonne, IL;Mathematics and Computer Science Division, Argonne National Laboratory, Argonne, IL
Venue:
Parallel Computing - Parallel data-intensive algorithms and applications
Year:
2002

Citing 9
Cited 132

The grid: blueprint for a new computing infrastructure

The grid: blueprint for a new computing infrastructure
LegionFS: a secure and scalable file system supporting cross-domain high-performance applications

Proceedings of the 2001 ACM/IEEE conference on Supercomputing
The SDSC storage resource broker

CASCON '98 Proceedings of the 1998 conference of the Centre for Advanced Studies on Collaborative research
A Network-Aware Distributed Storage Cache for Data Intensive Environments

HPDC '99 Proceedings of the 8th IEEE International Symposium on High Performance Distributed Computing
NetLogger: A Toolkit for Distributed System Performance Analysis

MASCOTS '00 Proceedings of the 8th International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems
On Individual and Aggregate TCP Performance

ICNP '99 Proceedings of the Seventh Annual International Conference on Network Protocols
The Kangaroo Approach to Data Movement on the Grid

HPDC '01 Proceedings of the 10th IEEE International Symposium on High Performance Distributed Computing
The Anatomy of the Grid: Enabling Scalable Virtual Organizations

International Journal of High Performance Computing Applications
Performance analysis in high-speed wide area IP-over-ATM network: top-to-bottom end-to-end monitoring

IEEE Network: The Magazine of Global Internetworking

Distributed computing research issues in grid computing

ACM SIGACT News
Grids, the TeraGrid, and Beyond

Computer
DYNAMO - DirectorY, Net Archiver and MOver

GRID '02 Proceedings of the Third International Workshop on Grid Computing
XML-Based Policy Engine Framework for Usage Policy Management in Grids

GRID '02 Proceedings of the Third International Workshop on Grid Computing
Applications of Virtual Data in the LIGO Experiment

PPAM '01 Proceedings of the th International Conference on Parallel Processing and Applied Mathematics-Revised Papers
Giggle: a framework for constructing scalable replica location services

Proceedings of the 2002 ACM/IEEE conference on Supercomputing
Design and Implementation of Multiple Addresses Parallel Transmission Architecture for Storage Area Network

MSS '03 Proceedings of the 20 th IEEE/11 th NASA Goddard Conference on Mass Storage Systems and Technologies (MSS'03)
Griz: experience with remote visualization over an optical grid

Future Generation Computer Systems - iGrid 2002
Quanta: a toolkit for high performance data delivery over photonic networks

Future Generation Computer Systems - iGrid 2002
Grid-enabled particle physics event analysis: experiences using a 10 Gb, high-latency network for a high-energy physics application

Future Generation Computer Systems - iGrid 2002
High-performance remote access to climate simulation data: a challenge problem for data grid technologies

Parallel Computing - Special issue: High performance computing with geographical data
Autonomous Con.guration of Grid Monitoring Systems

SAINT-W '04 Proceedings of the 2004 Symposium on Applications and the Internet-Workshops (SAINT 2004 Workshops)
References

Grid resource management
ZENTURIO: a grid middleware-based tool for experiment management of parallel and distributed applications

Journal of Parallel and Distributed Computing - Special issue on middleware
DIRAC: A Scalable Lightweight Architecture for High Throughput Computing

GRID '04 Proceedings of the 5th IEEE/ACM International Workshop on Grid Computing
High Performance Threaded Data Streaming for Large Scale Simulations

GRID '04 Proceedings of the 5th IEEE/ACM International Workshop on Grid Computing
Interactive Exploration of Large Remote Micro-CT Scans

VIS '04 Proceedings of the conference on Visualization '04
Experimental studies of data transport and data access of earth-science data over networks with high bandwidth delay products

Computer Networks: The International Journal of Computer and Telecommunications Networking - Special issue: Networking for the earth science
Grid -Based Parallel Data Streaming implemented for the Gyrokinetic Toroidal Code

Proceedings of the 2003 ACM/IEEE conference on Supercomputing
Self-Managing Sensor-Based Middleware for Performance Monitoring and Data Integration in Grids

IPDPS '05 Proceedings of the 19th IEEE International Parallel and Distributed Processing Symposium (IPDPS'05) - Papers - Volume 01
Adaptive Simulation: Dynamic Data Driven Application in Geophysical Mass Flows

IPDPS '05 Proceedings of the 19th IEEE International Parallel and Distributed Processing Symposium (IPDPS'05) - Papers - Volume 01
Modeling and Taming Parallel TCP on the Wide Area Network

IPDPS '05 Proceedings of the 19th IEEE International Parallel and Distributed Processing Symposium (IPDPS'05) - Papers - Volume 01
The Globus eXtensible Input/Output System (XIO): A Protocol Independent IO System for the Grid

IPDPS '05 Proceedings of the 19th IEEE International Parallel and Distributed Processing Symposium (IPDPS'05) - Workshop 4 - Volume 05
A Services Oriented Framework for Next Generation Data Analysis Centers

IPDPS '05 Proceedings of the 19th IEEE International Parallel and Distributed Processing Symposium (IPDPS'05) - Workshop 10 - Volume 11
Teraflows over Gigabit WANs with UDT

Future Generation Computer Systems - Special issue: High-speed networks and services for data-intensive grids: The DataTAG project
Quality of Service on the Grid Via Metascheduling with Resource Co-Scheduling and Co-Reservation

International Journal of High Performance Computing Applications
Virtual and Dynamic Hierarchical Architecture for E-Science Grid

International Journal of High Performance Computing Applications
Implementation and utilisation of a Grid-enabled problem solving environment in Matlab

Future Generation Computer Systems - Special section: Complex problem-solving environments for grid computing
Circus: Opportunistic Block Reordering for Scalable Content Servers

FAST '04 Proceedings of the 3rd USENIX Conference on File and Storage Technologies
GLARE: A Grid Activity Registration, Deployment and Provisioning Framework

SC '05 Proceedings of the 2005 ACM/IEEE conference on Supercomputing
The Globus Striped GridFTP Framework and Server

SC '05 Proceedings of the 2005 ACM/IEEE conference on Supercomputing
Multiple Job Scheduling in a Connection-Limited Data Parallel System

IEEE Transactions on Parallel and Distributed Systems
A taxonomy of Data Grids for distributed data sharing, management, and processing

ACM Computing Surveys (CSUR)
Implementation of a dynamic adjustment mechanism with efficient replica selection in data grid environments

Proceedings of the 2006 ACM symposium on Applied computing
Redundant parallel data transfer schemes for the grid environment

ACSW Frontiers '06 Proceedings of the 2006 Australasian workshops on Grid computing and e-research - Volume 54
Policy-Directed Data Movement in Grids

ICPADS '06 Proceedings of the 12th International Conference on Parallel and Distributed Systems - Volume 1
Data mining middleware for wide-area high-performance networks

Future Generation Computer Systems - IGrid 2005: The global lambda integrated facility
Grid capacity planning with negotiation-based advance reservation for optimized QoS

Proceedings of the 2006 ACM/IEEE conference on Supercomputing
Specification-correct and scalable coordination of Grid applications

Future Generation Computer Systems
Improvements on dynamic adjustment mechanism in co-allocation data grid environments

The Journal of Supercomputing
A resource broker with an efficient network information model on grid environments

The Journal of Supercomputing
Worldwide computing: Adaptive middleware and programming technology for dynamic Grid environments

Scientific Programming - Dynamic Grids and Worldwide Computing
Protecting Grid Data Transfer Services with Active Network Interfaces

GRID '05 Proceedings of the 6th IEEE/ACM International Workshop on Grid Computing
Partial content distribution on high performance networks

Proceedings of the 16th international symposium on High performance distributed computing
Direct-pNFS: scalable, transparent, and versatile access to parallel file systems

Proceedings of the 16th international symposium on High performance distributed computing
Implementing MPI-IO Atomic Mode and Shared File Pointers Using MPI One-Sided Communication

International Journal of High Performance Computing Applications
Application-layer multipath data transfer via TCP: Schemes and performance tradeoffs

Performance Evaluation
Performance enhancement of TCP in high-speed networks

Information Sciences: an International Journal
Exploring TCP Parallelisation for performance improvement in heterogeneous networks

Computer Communications
Data management services and transfer scheme in ChinaGrid

International Journal of Web and Grid Services
A collaborative experimentation environment for biosciences

International Journal of Networking and Virtual Organisations
A user-level secure grid file system

Proceedings of the 2007 ACM/IEEE conference on Supercomputing
Exploiting connector knowledge to efficiently disseminate highly voluminous data sets

Proceedings of the 3rd international workshop on Sharing and reusing architectural knowledge
Flexible and efficient transfer of multidimensional data in grid

CHINA HPC '07 Proceedings of the 2007 Asian technology information program's (ATIP's) 3rd workshop on High performance computing in China: solution approaches to impediments for high performance computing
File transfer in Grid applications at deployment, execution and retrieval

Multiagent and Grid Systems - Grid Computing, high performance and distributed applications
Improving GridFTP transfers by means of a multiagent parallel file system

Multiagent and Grid Systems - Grid Computing, high performance and distributed applications
Efficient reuse of replicated parallel data segments in computational grids

Future Generation Computer Systems
DIMM: a distributed metadata management for data-intensive HPC environments

DADC '08 Proceedings of the 2008 international workshop on Data-aware distributed computing
Performance of file replication policies for real-time file access in data grids

Proceedings of the first international conference on Networks for grid applications
A Data Management Framework for Urgent Geoscience Workflows

ICCS '08 Proceedings of the 8th international conference on Computational Science, Part II
Building Hierarchical Grid Storage Using the Gfarm Global File System and the JuxMem Grid Data-Sharing Service

Euro-Par '08 Proceedings of the 14th international Euro-Par conference on Parallel Processing
Experimenting the Query Performance of a Grid-Based Sensor Network Data Warehouse

Globe '08 Proceedings of the 1st international conference on Data Management in Grid and Peer-to-Peer Systems
Rethinking FTP: Aggressive block reordering for large file transfers

ACM Transactions on Storage (TOS)
Data Transformation Services over Grids with Real-Time Bound Constraints

OTM '08 Proceedings of the OTM 2008 Confederated International Conferences, CoopIS, DOA, GADA, IS, and ODBASE 2008. Part I on On the Move to Meaningful Internet Systems:
Design and implementation of a workflow-based resource broker with information system on computational grids

The Journal of Supercomputing
A grid middleware for data management exploiting peer-to-peer techniques

Future Generation Computer Systems
Distributed Management of Massive Data: An Efficient Fine-Grain Data Access Scheme

High Performance Computing for Computational Science - VECPAR 2008
Enhancement of anticipative recursively adjusting mechanism for redundant parallel file transfer in data grids

Journal of Network and Computer Applications
A Recursively-Adjusting Co-allocation scheme with a Cyber-Transformer in Data Grids

Future Generation Computer Systems
A study on performance of dynamic file replication algorithms for real-time file access in Data Grids

Future Generation Computer Systems
Use of competition detection in TCP for simple topology networks

Computer Networks: The International Journal of Computer and Telecommunications Networking
Montage: a grid portal and software toolkit for science-grade astronomical image mosaicking

International Journal of Computational Science and Engineering
Block-Based Concurrent and Storage-Aware Data Streaming for Grid Applications with Lots of Small Files

CCGRID '09 Proceedings of the 2009 9th IEEE/ACM International Symposium on Cluster Computing and the Grid
A Sharable Storage Service for Distributed Computing Systems in Combination of Remote and Local Storage

ICA3PP '09 Proceedings of the 9th International Conference on Algorithms and Architectures for Parallel Processing
On utilization of the grid computing technology for video conversion and 3D rendering

Computer Standards & Interfaces
Optimizing Data Management in Grid Environments

OTM '09 Proceedings of the Confederated International Conferences, CoopIS, DOA, IS, and ODBASE 2009 on On the Move to Meaningful Internet Systems: Part I
Intrusion Detection Using Neural Networks: A Grid Computing Based Data Mining Approach

ICONIP '09 Proceedings of the 16th International Conference on Neural Information Processing: Part II
Teraflows over Gigabit WANs with UDT

Future Generation Computer Systems - Special issue: High-speed networks and services for data-intensive grids: The DataTAG project
Implementation and utilisation of a Grid-enabled problem solving environment in Matlab

Future Generation Computer Systems - Special section: Complex problem-solving environments for grid computing
A high performance suite of data services for grids

Future Generation Computer Systems
Adaptive popularity-driven replica placement in hierarchical data grids

The Journal of Supercomputing
A new dynamic replication algorithm for European data grid

Proceedings of the Third Annual ACM Bangalore Conference
A high-performance virtual storage system for Taiwan UniGrid

GPC'07 Proceedings of the 2nd international conference on Advances in grid and pervasive computing
A novel data grid coherence protocol using pipeline-based aggressive copy method

GPC'07 Proceedings of the 2nd international conference on Advances in grid and pervasive computing
GRAVY: towards virtual file system for the grid

GPC'07 Proceedings of the 2nd international conference on Advances in grid and pervasive computing
Bioportal: a portal for deployment of bioinformatics applications on cluster and grid environments

VECPAR'06 Proceedings of the 7th international conference on High performance computing for computational science
A practical evaluation of a data consistency protocol for efficient visualization in grid applications

VECPAR'06 Proceedings of the 7th international conference on High performance computing for computational science
Virtual and dynamic hierarchical architecture for e-science grid

ICCS'03 Proceedings of the 2003 international conference on Computational science: PartIII
Fast parallel algorithm for volume rendering and its experiment on computational grid

ICCS'03 Proceedings of the 1st international conference on Computational science: PartI
Redundant parallel file transfer with anticipative recursively-adjusting scheme in data grids

ICA3PP'07 Proceedings of the 7th international conference on Algorithms and architectures for parallel processing
On the evaluation of gridification effort and runtime aspects of JGRIM applications

Future Generation Computer Systems
Towards secure data management system for grid environment based on the cell broadband engine

PPAM'07 Proceedings of the 7th international conference on Parallel processing and applied mathematics
Co-allocation in data grids: a global, multi-user perspective

GPC'08 Proceedings of the 3rd international conference on Advances in grid and pervasive computing
A dynamic adjustment strategy for file transformation in data grids

NPC'07 Proceedings of the 2007 IFIP international conference on Network and parallel computing
Grid computing: experiment management, tool integration, and scientific workflows

Grid computing: experiment management, tool integration, and scientific workflows
Update propagation technique for data grid

ICCSA'07 Proceedings of the 2007 international conference on Computational science and its applications - Volume Part III
Delay-based congestion avoidance for QoS provisioning in wired/wireless networks

ICC'09 Proceedings of the 2009 IEEE international conference on Communications
Implementation of a medical image file accessing system in co-allocation data grids

Future Generation Computer Systems
File replication, maintenance, and consistency management services in data grids

The Journal of Supercomputing
Dynamic replication in a data grid using a Modified BHR Region Based Algorithm

Future Generation Computer Systems
Implementation of a dynamic adjustment strategy for parallel file transfer in co-allocation data grids

The Journal of Supercomputing
IMAGE: an approach to building standards-based enterprise grids

IPDPS'06 Proceedings of the 20th international conference on Parallel and distributed processing
MPI-IO/L: efficient remote I/O for MPI-IO via logistical networking

IPDPS'06 Proceedings of the 20th international conference on Parallel and distributed processing
BlobSeer: Next-generation data management for large scale infrastructures

Journal of Parallel and Distributed Computing
Hybrid Computing-Where HPC meets grid and Cloud Computing

Future Generation Computer Systems
Circus: opportunistic block reordering for scalable content servers

FAST'04 Proceedings of the 3rd USENIX conference on File and storage technologies
Design of a hierarchical group to realize a scalable group

Journal of Mobile Multimedia
Improvement of the bandwidth of cross-site MPI communication using optical fiber

EuroMPI'11 Proceedings of the 18th European MPI Users' Group conference on Recent advances in the message passing interface
Gvu: a view-oriented framework for data management in grid environments

HiPC'06 Proceedings of the 13th international conference on High Performance Computing
Coordinated co-allocator model for data grid in multi-sender environment

ICSOC'06 Proceedings of the 4th international conference on Service-Oriented Computing
ClusteriX data management system and its integration with applications

PPAM'05 Proceedings of the 6th international conference on Parallel Processing and Applied Mathematics
On utilization of the grid computing technology for video conversion and 3d rendering

ISPA'05 Proceedings of the Third international conference on Parallel and Distributed Processing and Applications
A recursive-adjustment co-allocation scheme in data grid environments

ICA3PP'05 Proceedings of the 6th international conference on Algorithms and Architectures for Parallel Processing
GridFTP and parallel TCP support in naradabrokering

ICA3PP'05 Proceedings of the 6th international conference on Algorithms and Architectures for Parallel Processing
Replica based distributed metadata management in grid environment

ICCS'06 Proceedings of the 6th international conference on Computational Science - Volume Part IV
AME: an anyscale many-task computing engine

Proceedings of the 6th workshop on Workflows in support of large-scale science
An efficient network information model using NWS for grid computing environments

GCC'05 Proceedings of the 4th international conference on Grid and Cooperative Computing
Replica selection on co-allocation data grids

ISPA'04 Proceedings of the Second international conference on Parallel and Distributed Processing and Applications
An extensible framework for advanced distributed virtual environment on grid

AIS'04 Proceedings of the 13th international conference on AI, Simulation, and Planning in High Autonomy Systems
Grid-BGC: a grid-enabled terrestrial carbon cycle modeling system

Euro-Par'05 Proceedings of the 11th international Euro-Par conference on Parallel Processing
Grid file transfer during deployment, execution, and retrieval

ODBASE'06/OTM'06 Proceedings of the 2006 Confederated international conference on On the Move to Meaningful Internet Systems: CoopIS, DOA, GADA, and ODBASE - Volume Part II
Performance analysis of applying replica selection technology for data grid environments

PaCT'05 Proceedings of the 8th international conference on Parallel Computing Technologies
Examining TCP parallelization related methods for various packet losses

WWIC'05 Proceedings of the Third international conference on Wired/Wireless Internet Communications
Web service based data management for grid applications

EuroWeb'02 Proceedings of the 2002 international conference on EuroWeb
End-to-end quality of service for high-end applications

Computer Communications
Allocating replicas in large-scale data grids using a QoS-aware distributed technique with workload constraints

International Journal of Grid and Utility Computing
Energy conservative mobile cloud infrastructure

GPC'12 Proceedings of the 7th international conference on Advances in Grid and Pervasive Computing
Data transfer in advance on cluster

PaCT'07 Proceedings of the 9th international conference on Parallel Computing Technologies
Data Placement in P2P Data Grids Considering the Availability, Security, Access Performance and Load Balancing

Journal of Grid Computing
Job scheduling and dynamic data replication in data grid environment

The Journal of Supercomputing
Dynamic replica placement and selection strategies in data grids- A comprehensive survey

Journal of Parallel and Distributed Computing
Efficient and robust large medical image retrieval in mobile cloud computing environment

Information Sciences: an International Journal

Quantified Score

Hi-index	0.00

Visualization

Abstract

Applications that explore, query, analyze, visualize, and, in general, process very large scale data sets are known as Data Intensive Applications. Large scale data intensive computing plays an increasingly important role in many scientific activities ...