Scripting distributed scientific workflows using Weaver

Authors:
Peter Bui;Li Yu;Andrew Thrasher;Rory Carmichael;Irena Lanc;Patrick Donnelly;Douglas Thain
Affiliations:
Department of Computer Science and Engineering, University of Notre Dame, Notre Dame, INUSA;Department of Computer Science and Engineering, University of Notre Dame, Notre Dame, INUSA;Department of Computer Science and Engineering, University of Notre Dame, Notre Dame, INUSA;Department of Computer Science and Engineering, University of Notre Dame, Notre Dame, INUSA;Department of Computer Science and Engineering, University of Notre Dame, Notre Dame, INUSA;Department of Computer Science and Engineering, University of Notre Dame, Notre Dame, INUSA;Department of Computer Science and Engineering, University of Notre Dame, Notre Dame, INUSA
Venue:
Concurrency and Computation: Practice & Experience
Year:
2012

Citing 19
Cited 0

A notation and system for expressing and executing cleanly typed workflows on messy scientific data

ACM SIGMOD Record
A taxonomy of scientific workflow systems for grid computing

ACM SIGMOD Record
LINQ: reconciling object, relations and XML in the .NET framework

Proceedings of the 2006 ACM SIGMOD international conference on Management of data
Automatic Grid workflow based on imperative programming languages: Research Articles

Concurrency and Computation: Practice & Experience - Workflow in Grid Systems
Scientific workflow management and the Kepler system: Research Articles

Concurrency and Computation: Practice & Experience - Workflow in Grid Systems
Pegasus: A framework for mapping complex scientific workflows onto distributed systems

Scientific Programming
Interpreting the data: Parallel analysis with Sawzall

Scientific Programming - Dynamic Grids and Worldwide Computing
MapReduce: simplified data processing on large clusters

OSDI'04 Proceedings of the 6th conference on Symposium on Opearting Systems Design & Implementation - Volume 6
Dryad: distributed data-parallel programs from sequential building blocks

Proceedings of the 2nd ACM SIGOPS/EuroSys European Conference on Computer Systems 2007
Pig latin: a not-so-foreign language for data processing

Proceedings of the 2008 ACM SIGMOD international conference on Management of data
SCOPE: easy and efficient parallel processing of massive data sets

Proceedings of the VLDB Endowment
Distributed data-parallel computing using a high-level programming language

Proceedings of the 2009 ACM SIGMOD International Conference on Management of data
Experience with BXGrid: a data repository and computing grid for biometrics research

Cluster Computing
Scientific workflow: a survey and research directions

PPAM'07 Proceedings of the 7th international conference on Parallel processing and applied mathematics
FlumeJava: easy, efficient data-parallel pipelines

PLDI '10 Proceedings of the 2010 ACM SIGPLAN conference on Programming language design and implementation
Harnessing parallelism in multicore clusters with the All-Pairs, Wavefront, and Makeflow abstractions

Cluster Computing
Towards long term data quality in a large scale biometrics experiment

Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing
Weaver: integrating distributed computing abstractions into scientific workflows using Python

Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing
ROARS: a scalable repository for data intensive scientific computing

Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing

Quantified Score

Hi-index	0.00

Visualization

Abstract

Weaver is a high-level distributed computing framework that enables researchers to construct scalable scientific data-processing workflows. Instead of developing a new workflow language, we introduce a domain-specific language built on top of Python called Weaver, which takes advantage of users' familiarity with the programming language, minimizes barriers to adoption, and allows for integration with a rich ecosystem of existing software. In this paper, we provide an overview of Weaver's programming model, which allows users to organize and specify scientific workflows by using a collection of datasets, functions, and abstractions. We also explain how these workflow specifications are compiled into a directed acyclic graph that is used by the Makeflow workflow manager to dispatch work to a variety of distributed execution platforms. To demonstrate the power and benefits of using the framework in constructing scientific research applications, the paper examines four distinct real-world applications scripted using Weaver and analyzes the performance, scalability, and impact of the distributed generated scientific workflows. Copyright © 2011 John Wiley & Sons, Ltd.