A notation and system for expressing and executing cleanly typed workflows on messy scientific data

Authors:
Yong Zhao;Jed Dobson;Ian Foster;Luc Moreau;Michael Wilde
Affiliations:
University of Chicago, Chicago, IL;Dartmouth College, Hanover, NH;University of Chicago, Chicago, IL;University of Southampton, Southampton, U.K.;Argonne National Laboratory, Argonne, IL
Venue:
ACM SIGMOD Record
Year:
2005

Citing 5
Cited 13

Chimera: AVirtual Data System for Representing, Querying, and Automating Data Derivation

SSDBM '02 Proceedings of the 14th International Conference on Scientific and Statistical Database Management
Kepler: An Extensible System for Design and Execution of Scientific Workflows

SSDBM '04 Proceedings of the 16th International Conference on Scientific and Statistical Database Management
Taverna: a tool for the composition and enactment of bioinformatics workflows

Bioinformatics
Programming scientific and distributed workflow with Triana services: Research Articles

Concurrency and Computation: Practice & Experience - Workflow in Grid Systems
XDTM: the XML data type and mapping for specifying datasets

EGC'05 Proceedings of the 2005 European conference on Advances in Grid Computing

Neuroimaging Data Provenance Using the LONI Pipeline Workflow Environment

Provenance and Annotation of Data and Processes
Harnessing parallelism in multicore clusters with the all-pairs and wavefront abstractions

Proceedings of the 18th ACM international symposium on High performance distributed computing
Experience with BXGrid: a data repository and computing grid for biometrics research

Cluster Computing
The next 700 data description languages

Journal of the ACM (JACM)
A formal model of dataflow repositories

DILS'07 Proceedings of the 4th international conference on Data integration in the life sciences
Parallelizing XML data-streaming workflows via MapReduce

Journal of Computer and System Sciences
Weaver: integrating distributed computing abstractions into scientific workflows using Python

Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing
Modeling and storing scientific protocols

OTM'06 Proceedings of the 2006 international conference on On the Move to Meaningful Internet Systems: AWeSOMe, CAMS, COMINF, IS, KSinBIT, MIOS-CIAO, MONET - Volume Part I
Applying the virtual data provenance model

IPAW'06 Proceedings of the 2006 international conference on Provenance and Annotation of Data
Scripting distributed scientific workflows using Weaver

Concurrency and Computation: Practice & Experience
Using relative costs in workflow scheduling to cope with input data uncertainty

Proceedings of the 10th International Workshop on Middleware for Grids, Clouds and e-Science
Designing and Deploying a Scientific Computing Cloud Platform

GRID '12 Proceedings of the 2012 ACM/IEEE 13th International Conference on Grid Computing
Stochastic DAG scheduling using a Monte Carlo approach

Journal of Parallel and Distributed Computing

Quantified Score

Hi-index	0.00

Visualization

Abstract

The description, composition, and execution of even logically simple scientific workflows are often complicated by the need to deal with "messy" issues like heterogeneous storage formats and ad-hoc file system structures. We show how these difficulties can be overcome via a typed, compositional workflow notation within which issues of physical representation are cleanly separated from logical typing, and by the implementation of this notation within the context of a powerful runtime system that supports distributed execution. The resulting notation and system are capable both of expressing complex workflows in a simple, compact form, and of enacting those workflows in distributed environments. We apply our technique to cognitive neuroscience workflows that analyze functional MRI image data, and demonstrate significant reductions in code size relative to other approaches.