Online aggregation and continuous query support in MapReduce

  • Authors:

  • Tyson Condie;Neil Conway;Peter Alvaro;Joseph M. Hellerstein;John Gerth;Justin Talbot;Khaled Elmeleegy;Russell Sears

  • Affiliations:

  • University of California at Berkeley, Berkeley, CA, USA;University of California at Berkeley, Berkeley, CA, USA;University of California at Berkeley, Berkeley, CA, USA;University of California at Berkeley, Berkeley, CA, USA;Stanford University, Stanford, CA, USA;Stanford University, Stanford, CA, USA;Yahoo! Research, Sunnyvale, CA, USA;Yahoo! Research, Sunnyvale, CA, USA

  • Venue:
  • Proceedings of the 2010 ACM SIGMOD International Conference on Management of data
  • Year:
  • 2010

Quantified Score

Hi-index 0.00

Visualization

Abstract

MapReduce is a popular framework for data-intensive distributed computing of batch jobs. To simplify fault tolerance, the output of each MapReduce task and job is materialized to disk before it is consumed. In this demonstration, we describe a modified MapReduce architecture that allows data to be pipelined between operators. This extends the MapReduce programming model beyond batch processing, and can reduce completion times and improve system utilization for batch jobs as well. We demonstrate a modified version of the Hadoop MapReduce framework that supports online aggregation, which allows users to see "early returns" from a job as it is being computed. Our Hadoop Online Prototype (HOP) also supports continuous queries, which enable MapReduce programs to be written for applications such as event monitoring and stream processing. HOP retains the fault tolerance properties of Hadoop, and can run unmodified user-defined MapReduce programs.