Challenges and approaches for distributed workflow-driven analysis of large-scale biological data: vision paper

Authors:
Ilkay Altintas;Jianwu Wang;Daniel Crawl;Weizhong Li
Affiliations:
University of California, San Diego, La Jolla, CA;University of California, San Diego, La Jolla, CA;University of California, San Diego, La Jolla, CA;University of California, San Diego, La Jolla, CA
Venue:
Proceedings of the 2012 Joint EDBT/ICDT Workshops
Year:
2012

Citing 19
Cited 1

Kepler: An Extensible System for Design and Execution of Scientific Workflows

SSDBM '04 Proceedings of the 16th International Conference on Scientific and Statistical Database Management
Taverna: a tool for the composition and enactment of bioinformatics workflows

Bioinformatics
Scientific workflow management and the Kepler system: Research Articles

Concurrency and Computation: Practice & Experience - Workflow in Grid Systems
Introduction and evaluation of Martlet: a scientific workflow language for abstracted parallelisation

Proceedings of the 16th international conference on World Wide Web
MapReduce: simplified data processing on large clusters

Communications of the ACM - 50th anniversary issue: 1958 - 2008
Advanced data flow support for scientific grid workflow applications

Proceedings of the 2007 ACM/IEEE conference on Supercomputing
Data-Intensive Computing in the 21st Century

Computer
Composing Different Models of Computation in Kepler and Ptolemy II

ICCS '07 Proceedings of the 7th international conference on Computational Science, Part III: ICCS 2007
Heterogeneous composition of models of computation

Future Generation Computer Systems
CloudBurst

Bioinformatics
Fast and accurate short read alignment with Burrows–Wheeler transform

Bioinformatics
A MapReduce-Enabled Scientific Workflow Composition Framework

ICWS '09 Proceedings of the 2009 IEEE International Conference on Web Services
Accelerating Parameter Sweep Workflows by Utilizing Ad-hoc Network Computing Resources: An Ecological Example

SERVICES '09 Proceedings of the 2009 Congress on Services - I
Kepler + Hadoop: a general architecture facilitating data-intensive applications in scientific workflow systems

Proceedings of the 4th Workshop on Workflows in Support of Large-Scale Science
All-Pairs: An Abstraction for Data-Intensive Computing on Campus Grids

IEEE Transactions on Parallel and Distributed Systems
Nephele/PACTs: a programming model and execution framework for web-scale analytical processing

Proceedings of the 1st ACM symposium on Cloud computing
A fault-tolerance architecture for Kepler-based distributed scientific workflows

SSDBM'10 Proceedings of the 22nd international conference on Scientific and statistical database management
Exploiting Dynamic Resource Allocation for Efficient Parallel Data Processing in the Cloud

IEEE Transactions on Parallel and Distributed Systems
Provenance collection support in the kepler scientific workflow system

IPAW'06 Proceedings of the 2006 international conference on Provenance and Annotation of Data

Approaches to Distributed Execution of Scientific Workflows in Kepler

Fundamenta Informaticae - Scalable Workflow Enactment Engines and Technology

Quantified Score

Hi-index	0.00

Visualization

Abstract

Next-generation DNA sequencing machines are generating a very large amount of sequence data with applications in many scientific challenges and placing unprecedented demands on traditional single-processor bioinformatics algorithms. Middleware and technologies for scientific workflows and data-intensive computing promise new capabilities to enable rapid analysis of next-generation sequence data. Based on this motivation and our previous experiences in bioinformatics and distributed scientific workflows, we are creating a Kepler Scientific Workflow System module, called "bioKepler", that facilitates the development of Kepler workflows for integrated execution of bioinformatics applications in distributed environments. This vision paper discusses the challenges related to next-generation sequencing data, explains the approaches taken in bioKepler to help with analysis of such data, and presents preliminary results demonstrating these approaches.