Data management and transfer in high-performance computational grid environments
Parallel Computing - Parallel data-intensive algorithms and applications
An end-to-end approach to globally scalable network storage
Proceedings of the 2002 conference on Applications, technologies, architectures, and protocols for computer communications
Improving MPI-IO Output Performance with Active Buffering Plus Threads
IPDPS '03 Proceedings of the 17th International Symposium on Parallel and Distributed Processing
DataMover: Robust Terabyte-Scale Multi-file Replication over Wide-Area Networks
SSDBM '04 Proceedings of the 16th International Conference on Scientific and Statistical Database Management
Performance and Scalability of a Replica Location Service
HPDC '04 Proceedings of the 13th IEEE International Symposium on High Performance Distributed Computing
Grid -Based Parallel Data Streaming implemented for the Gyrokinetic Toroidal Code
Proceedings of the 2003 ACM/IEEE conference on Supercomputing
Remote Visualization by Browsing Image Based Databases with Logistical Networking
Proceedings of the 2003 ACM/IEEE conference on Supercomputing
A modeling and executive environment for distributed scientific workflows
SSDBM '03 Proceedings of the 15th International Conference on Scientific and Statistical Database Management
An Autonomic Service Architecture for Self-Managing Grid Applications
GRID '05 Proceedings of the 6th IEEE/ACM International Workshop on Grid Computing
Workflow automation for processing plasma fusion simulation data
Proceedings of the 2nd workshop on Workflows in support of large-scale science
A Self-Managing Wide-Area Data Streaming Service using Model-based Online Control
GRID '06 Proceedings of the 7th IEEE/ACM International Conference on Grid Computing
CCGRID '09 Proceedings of the 2009 9th IEEE/ACM International Symposium on Cluster Computing and the Grid
Enabling efficient and flexible coupling of parallel scientific applications
IPDPS'06 Proceedings of the 20th international conference on Parallel and distributed processing
A case for MapReduce over the internet
Proceedings of the 2013 ACM Cloud and Autonomic Computing Conference
| Hi-index | 0.00 |
We have developed a threaded parallel data streaming approach using Logistical Networking (LN) to transfer multi-terabyte simulation data from computers at NERSC to our local analysis/visualization cluster, as the simulation executes, with negligible overhead. Data transfer experiments show that this concurrent data transfer approach is more favorable compared with writing to local disk and later transferring this data to be post-processed. Our algorithms are network aware, and can stream data at up to 97Mbs on a 100Mbs link from CA to NJ during a live simulation, using less than 5% CPU overhead at NERSC. This method is the first step in setting up a pipeline for simulation workflow and data management.