High Performance Threaded Data Streaming for Large Scale Simulations
GRID '04 Proceedings of the 5th IEEE/ACM International Workshop on Grid Computing
The performance analysis of linux networking - Packet receiving
Computer Communications
Archive migration through workflow automation
PDCS '07 Proceedings of the 19th IASTED International Conference on Parallel and Distributed Computing and Systems
A DSM-based fragmented data sharing framework for grids
Future Generation Computer Systems
RRS: replica registration service for data grids
DMG 2005 Proceedings of the First VLDB conference on Data Management in Grids
Grid-BGC: a grid-enabled terrestrial carbon cycle modeling system
Euro-Par'05 Proceedings of the 11th international Euro-Par conference on Parallel Processing
| Hi-index | 0.00 |
Typically, large scientific datasets (order of terabytes)are generated at large computational centers,and stored on mass storage systems. However, largesubsets of the data need to be moved to facilitiesavailable to application scientists for analysis. Filereplication of thousands of files is a tedious, errorprone, but extremely important task in scientificapplications. The automation of the file replicationtask requires automatic space acquisition and reuse,and monitoring the progress of staging thousands offiles from the source mass storage system, transferringthem over the network, archiving them at thetarget mass storage system or disk systems, andrecovering from transient system failures. We havedeveloped a robust replication system, calledDataMover, which is now in regular use in High-Energy-Physics and Climate modeling experiments. Only a single command is necessary to request multi-filereplication or the replication of an entiredirectory. A web-based tool was developed todynamically monitor the progress of the multi-filereplication process.