Implementing Parallel Google Map-Reduce in Eden

  • Authors:
  • Jost Berthold;Mischa Dieterle;Rita Loogen

  • Affiliations:
  • Fachbereich Mathematik und Informatik Hans Meerwein Straße, Philipps-Universität Marburg, Marburg, Germany D-35032;Fachbereich Mathematik und Informatik Hans Meerwein Straße, Philipps-Universität Marburg, Marburg, Germany D-35032;Fachbereich Mathematik und Informatik Hans Meerwein Straße, Philipps-Universität Marburg, Marburg, Germany D-35032

  • Venue:
  • Euro-Par '09 Proceedings of the 15th International Euro-Par Conference on Parallel Processing
  • Year:
  • 2009

Quantified Score

Hi-index 0.00

Visualization

Abstract

Recent publications have emphasised map-reduce as a general programming model (labelled Google map-reduce), and described existing high-performance implementations for large data sets. We present two parallel implementations for this Google map-reduce skeleton, one following earlier work, and one optimised version, in the parallel Haskell extension Eden. Eden's specific features, like lazy stream processing, dynamic reply channels, and nondeterministic stream merging, support the efficient implementation of the complex coordination structure of this skeleton. We compare the two implementations of the Google map-reduce skeleton in usage and performance, and deliver runtime analyses for example applications. Although very flexible, the Google map-reduce skeleton is often too general, and typical examples reveal a better runtime behaviour using alternative skeletons.