Parallelizing XML data-streaming workflows via MapReduce

  • Authors:

  • Daniel Zinn;Shawn Bowers;Sven Köhler;Bertram Ludäscher

  • Affiliations:

  • Department of Computer Science, University of California, Davis, United States;UC Davis Genome Center, University of California, Davis, United States and Department of Computer Science, Gonzaga University, United States;UC Davis Genome Center, University of California, Davis, United States;Department of Computer Science, University of California, Davis, United States and UC Davis Genome Center, University of California, Davis, United States

  • Venue:
  • Journal of Computer and System Sciences
  • Year:
  • 2010

Quantified Score

Hi-index 0.00

Visualization

Abstract

In prior work it has been shown that the design of scientific workflows can benefit from a collection-oriented modeling paradigm which views scientific workflows as pipelines of XML stream processors. In this paper, we present approaches for exploiting data parallelism in XML processing pipelines through novel compilation strategies to the MapReduce framework. Pipelines in our approach consist of sequences of processing steps that receive XML-structured data and produce, often through calls to ''black-box'' (scientific) functions, modified (i.e., updated) XML structures. Our main contributions are (i) the development of a set of strategies for compiling scientific workflows, modeled as XML processing pipelines, into parallel MapReduce networks, and (ii) a discussion of their advantages and trade-offs, based on a thorough experimental evaluation of the various translation strategies. Our evaluation uses the Hadoop MapReduce system as an implementation platform. Our results show that execution times of XML workflow pipelines can be significantly reduced using our compilation strategies. These efficiency gains, together with the benefits of MapReduce (e.g., fault tolerance) make our approach ideal for executing large-scale, compute-intensive XML-based scientific workflows.