Grid -Based Parallel Data Streaming implemented for the Gyrokinetic Toroidal Code

Authors:
S. Klasky;S. Ethier;Z. Lin;K. Martins;D. McCune;R. Samtaney
Affiliations:
Princeton University, NJ;Princeton University, NJ;University of California, Irvine;Princeton University, NJ;Princeton University, NJ;Princeton University, NJ
Venue:
Proceedings of the 2003 ACM/IEEE conference on Supercomputing
Year:
2003

Citing 8
Cited 9

Gyrokinetic particle simulation model

Journal of Computational Physics
Partially linearized algorithms in gyrokinetic particle simulation

Journal of Computational Physics
Computational grids

The grid
Data management and transfer in high-performance computational grid environments

Parallel Computing - Parallel data-intensive algorithms and applications
Processing large-scale multi-dimensional data in parallel and distributed environments

Parallel Computing - Parallel data-intensive algorithms and applications
Design of a Framework for Data-Intensive Wide-Area Applications

HCW '00 Proceedings of the 9th Heterogeneous Computing Workshop
GridMapper: A Tool for Visualizing the Behavior of Large-Scale Distributed Systems

HPDC '02 Proceedings of the 11th IEEE International Symposium on High Performance Distributed Computing
Improving MPI-IO Output Performance with Active Buffering Plus Threads

IPDPS '03 Proceedings of the 17th International Symposium on Parallel and Distributed Processing

High Performance Threaded Data Streaming for Large Scale Simulations

GRID '04 Proceedings of the 5th IEEE/ACM International Workshop on Grid Computing
Visualizing Gyrokinetic Simulations

VIS '04 Proceedings of the conference on Visualization '04
Noncontiguous locking techniques for parallel file systems

Proceedings of the 2007 ACM/IEEE conference on Supercomputing
FaReS: Fair Resource Scheduling for VMM-Bypass InfiniBand Devices

CCGRID '10 Proceedings of the 2010 10th IEEE/ACM International Conference on Cluster, Cloud and Grid Computing
Managing Variability in the IO Performance of Petascale Storage Systems

Proceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis
Just in time: adding value to the IO pipelines of high performance applications with JITStaging

Proceedings of the 20th international symposium on High performance distributed computing
Characterizing output bottlenecks in a supercomputer

SC '12 Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis
Data deduplication in a hybrid architecture for improving write performance

Proceedings of the 3rd International Workshop on Runtime and Operating Systems for Supercomputers
GoldRush: resource efficient in situ scientific data analytics using fine-grained interference aware execution

SC '13 Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis

Quantified Score

Hi-index	0.00

Visualization

Abstract

We have developed a threaded parallel data streaming approach using Globus to transfer multi-terabyte simulation data from a remote supercomputer to the scientistýs home analysis/visualization cluster, as the simulation executes, with negligible overhead. Data transfer experiments show that this concurrent data transfer approach is more favorable compared with writing to local disk and then transferring this data to be post-processed. The present approach is conducive to using the grid to pipeline the simulation with post-processing and visualization. We have applied this method to the Gyrokinetic Toroidal Code (GTC), a 3-dimensional particle-in-cell code used to study micro-turbulence in magnetic confinement fusion from first principles plasma theory.