Dataflow coordination of data-parallel tasks via MPI 3.0

Authors:
Justin M. Wozniak;Tom Peterka;Timothy G. Armstrong;James Dinan;Ewing Lusk;Michael Wilde;Ian Foster
Affiliations:
Argonne National Laboratory, Argonne, IL;Argonne National Laboratory, Argonne, IL;University of Chicago, Chicago, IL;Argonne National Laboratory, Argonne, IL;Argonne National Laboratory, Argonne, IL;Argonne National Laboratory, Argonne, IL;Argonne National Laboratory, Argonne, IL
Venue:
Proceedings of the 20th European MPI Users' Group Meeting
Year:
2013

Citing 10
Cited 0

Adaptable, metadata rich IO methods for portable high performance IO

IPDPS '09 Proceedings of the 2009 IEEE International Symposium on Parallel&Distributed Processing
A configurable algorithm for parallel image-compositing applications

Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis
Noncollective communicator creation in MPI

EuroMPI'11 Proceedings of the 18th European MPI Users' Group conference on Recent advances in the message passing interface
A Study of Parallel Particle Tracing for Steady-State and Time-Varying Flow Fields

IPDPS '11 Proceedings of the 2011 IEEE International Parallel & Distributed Processing Symposium
Coasters: Uniform Resource Provisioning and Access for Clouds and Grids

UCC '11 Proceedings of the 2011 Fourth IEEE International Conference on Utility and Cloud Computing
Swift: A language for distributed parallel scripting

Parallel Computing
Load Balancing of Dynamical Nucleation Theory Monte Carlo Simulations through Resource Sharing Barriers

IPDPS '12 Proceedings of the 2012 IEEE 26th International Parallel and Distributed Processing Symposium
Efficient multithreaded context ID allocation in MPI

EuroMPI'12 Proceedings of the 19th European conference on Recent Advances in the Message Passing Interface
Versatile communication algorithms for data analysis

EuroMPI'12 Proceedings of the 19th European conference on Recent Advances in the Message Passing Interface
Turbine: a distributed-memory dataflow engine for extreme-scale many-task applications

Proceedings of the 1st ACM SIGMOD Workshop on Scalable Workflow Execution Engines and Technologies

Quantified Score

Hi-index	0.00

Visualization

Abstract

Scientific applications are often complex collections of many large-scale tasks. Mature tools exist for describing task-parallel workflows consisting of serial tasks, and a variety of tools exist for programming a single data-parallel operation. However, few tools cover the intersection of these two models. In this work, we extend the load balancing library ADLB to support parallel tasks. We demonstrate how applications can easily be composed of parallel tasks using Swift dataflow scripts, which are compiled to ADLB programs with performance comparable to hand-coded equivalents. By combining this framework with data-parallel analysis libraries, we are able to dynamically execute many instances of a parallel data analysis application in support of a parameter exploration workload.