High end scientific codes with computational I/O pipelines: improving their end-to-end performance

Authors:
Fang Zheng;Jianting Cao;Jai Dayal;Greg Eisenhauer;Karsten Schwan;Matthew Wolf;Hasan Abbasi;Scott Klasky;Norbert Podhorszki
Affiliations:
Georgia Institute of Technology, Atlanta, GA, USA;Georgia Institute of Technology, Atlanta, GA, USA;Georgia Institute of Technology, Atlanta, GA, USA;Georgia Institute of Technology, Atlanta, GA, USA;Georgia Institute of Technology, Atlanta, GA, USA;Georgia Institute of Technology, Atlanta, GA, USA;Oak Ridge National Laboratory, Oak Ridge, TN, USA;Oak Ridge National Laboratory, Oak Ridge, TN, USA;Oak Ridge National Laboratory, Oak Ridge, TN, USA
Venue:
Proceedings of the 2nd international workshop on Petascal data analytics: challenges and opportunities
Year:
2011

Citing 21
Cited 0

XChange: coupling parallel applications in a dynamic environment

CLUSTER '04 Proceedings of the 2004 IEEE International Conference on Cluster Computing
Scientific workflow management and the Kepler system: Research Articles

Concurrency and Computation: Practice & Experience - Workflow in Grid Systems
Remote runtime steering of integrated terascale simulation and visualization

Proceedings of the 2006 ACM/IEEE conference on Supercomputing
Pegasus: A framework for mapping complex scientific workflows onto distributed systems

Scientific Programming
Modeling the Impact of Checkpoints on Next-Generation Systems

MSST '07 Proceedings of the 24th IEEE Conference on Mass Storage Systems and Technologies
Data Management Challenges of Data-Intensive Scientific Workflows

CCGRID '08 Proceedings of the 2008 Eighth IEEE International Symposium on Cluster Computing and the Grid
DataStager: scalable data staging services for petascale applications

Proceedings of the 18th ACM international symposium on High performance distributed computing
Adaptable, metadata rich IO methods for portable high performance IO

IPDPS '09 Proceedings of the 2009 IEEE International Symposium on Parallel&Distributed Processing
Plasma fusion code coupling using scalable I/O services and scientific workflows

Proceedings of the 4th Workshop on Workflows in Support of Large-Scale Science
GePSeA: A General-Purpose Software Acceleration Framework for Lightweight Task Offloading

ICPP '09 Proceedings of the 2009 International Conference on Parallel Processing
Managing scientific data

Communications of the ACM
Extreme Scaling of Production Visualization Software on Diverse Architectures

IEEE Computer Graphics and Applications
In Situ Visualization for Large-Scale Combustion Simulations

IEEE Computer Graphics and Applications
DataSpaces: an interaction and coordination framework for coupled simulation workflows

Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing
Managing Variability in the IO Performance of Petascale Storage Systems

Proceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis
Just in time: adding value to the IO pipelines of high performance applications with JITStaging

Proceedings of the 20th international symposium on High performance distributed computing
Six degrees of scientific data: reading patterns for extreme scale science IO

Proceedings of the 20th international symposium on High performance distributed computing
A flexible architecture integrating monitoring and analytics for managing large-scale data centers

Proceedings of the 8th ACM international conference on Autonomic computing
Enabling Multi-physics Coupled Simulations within the PGAS Programming Framework

CCGRID '11 Proceedings of the 2011 11th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing
Parallel computational steering and analysis for HPC applications using a paraview interface and the HDF5 DSM virtual file driver

EG PGV'11 Proceedings of the 11th Eurographics conference on Parallel Graphics and Visualization
Parallel in situ coupling of simulation with a fully featured visualization system

EG PGV'11 Proceedings of the 11th Eurographics conference on Parallel Graphics and Visualization

Quantified Score

Hi-index	0.00

Visualization

Abstract

This paper uses computational I/O pipelines to integrate computations into the I/O path that perform data analytics on the data generated by scientific simulations. A novel attribute is the use of a pluggable execution environment in which analysis tools can be orchestrated into a multi-stage pipeline for processing simulation output data. Performance considerations are addressed through the use of a high performance data transport. The approach is evaluated with the end-to-end performance of a Magnetohydrodynamics application at large scale.