CIEL: a universal execution engine for distributed data-flow computing

Authors:
Derek G. Murray;Malte Schwarzkopf;Christopher Smowton;Steven Smith;Anil Madhavapeddy;Steven Hand
Affiliations:
University of Cambridge, Computer Laboratory;University of Cambridge, Computer Laboratory;University of Cambridge, Computer Laboratory;University of Cambridge, Computer Laboratory;University of Cambridge, Computer Laboratory;University of Cambridge, Computer Laboratory
Venue:
Proceedings of the 8th USENIX conference on Networked systems design and implementation
Year:
2011

Citing 31
Cited 47

MULTILISP: a language for concurrent symbolic computation

ACM Transactions on Programming Languages and Systems (TOPLAS)
Parallel database systems: the future of high performance database systems

Communications of the ACM
Programming parallel algorithms

Communications of the ACM
SQL: 1999, formerly known as SQL3

ACM SIGMOD Record
Scheduling multithreaded computations by work stealing

Journal of the ACM (JACM)
Can programming be liberated from the von Neumann style?: a functional style and its algebra of programs

Communications of the ACM
Universality of data retrieval languages

POPL '79 Proceedings of the 6th ACM SIGACT-SIGPLAN symposium on Principles of programming languages
What You Always Wanted to Know About Datalog (And Never Dared to Ask)

IEEE Transactions on Knowledge and Data Engineering
The Design and Implementation of Glasgow Distributed Haskell

IFL '00 Selected Papers from the 12th International Workshop on Implementation of Functional Languages
MPI: A Message-Passing Interface Standard

MPI: A Message-Passing Interface Standard
The Google file system

SOSP '03 Proceedings of the nineteenth ACM symposium on Operating systems principles
X10: an object-oriented approach to non-uniform cluster computing

OOPSLA '05 Proceedings of the 20th annual ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications
MapReduce: simplified data processing on large clusters

OSDI'04 Proceedings of the 6th conference on Symposium on Opearting Systems Design & Implementation - Volume 6
Adaptive and reliable parallel computing on networks of workstations

ATEC '97 Proceedings of the annual conference on USENIX Annual Technical Conference
Dryad: distributed data-parallel programs from sequential building blocks

Proceedings of the 2nd ACM SIGOPS/EuroSys European Conference on Computer Systems 2007
Pig latin: a not-so-foreign language for data processing

Proceedings of the 2008 ACM SIGMOD international conference on Management of data
SCOPE: easy and efficient parallel processing of massive data sets

Proceedings of the VLDB Endowment
MapReduce for Data Intensive Scientific Analyses

ESCIENCE '08 Proceedings of the 2008 Fourth IEEE International Conference on eScience
The Design of OpenMP Tasks

IEEE Transactions on Parallel and Distributed Systems
Quincy: fair scheduling for distributed computing clusters

Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles
Hadoop: The Definitive Guide

Hadoop: The Definitive Guide
Boom analytics: exploring data-centric, declarative programming for the cloud

Proceedings of the 5th European conference on Computer systems
Delay scheduling: a simple technique for achieving locality and fairness in cluster scheduling

Proceedings of the 5th European conference on Computer systems
Pregel: a system for large-scale graph processing

Proceedings of the 2010 ACM SIGMOD International Conference on Management of data
MapReduce online

NSDI'10 Proceedings of the 7th USENIX conference on Networked systems design and implementation
DryadLINQ: a system for general-purpose distributed data-parallel computing using a high-level language

OSDI'08 Proceedings of the 8th USENIX conference on Operating systems design and implementation
Scripting the cloud with skywriting

HotCloud'10 Proceedings of the 2nd USENIX conference on Hot topics in cloud computing
HaLoop: efficient iterative data processing on large clusters

Proceedings of the VLDB Endowment
Nectar: automatic management of data and computation in datacenters

OSDI'10 Proceedings of the 9th USENIX conference on Operating systems design and implementation
Piccolo: building fast, distributed programs with partitioned tables

OSDI'10 Proceedings of the 9th USENIX conference on Operating systems design and implementation
Mesos: a platform for fine-grained resource sharing in the data center

Proceedings of the 8th USENIX conference on Networked systems design and implementation

Mesos: a platform for fine-grained resource sharing in the data center

Proceedings of the 8th USENIX conference on Networked systems design and implementation
Non-deterministic parallelism considered useful

HotOS'13 Proceedings of the 13th USENIX conference on Hot topics in operating systems
Managing data transfers in computer clusters with orchestra

Proceedings of the ACM SIGCOMM 2011 conference
Disco: a computing platform for large-scale data analytics

Proceedings of the 10th ACM SIGPLAN workshop on Erlang
Towards Haskell in the cloud

Proceedings of the 4th ACM symposium on Haskell
DOT: a matrix model for analyzing, optimizing and deploying software for big data analytics in distributed systems

Proceedings of the 2nd ACM Symposium on Cloud Computing
PrIter: a distributed framework for prioritized iterative computations

Proceedings of the 2nd ACM Symposium on Cloud Computing
Programming micro-aerial vehicle swarms with karma

Proceedings of the 9th ACM Conference on Embedded Networked Sensor Systems
Jockey: guaranteed job latency in data parallel clusters

Proceedings of the 7th ACM european conference on Computer Systems
MadLINQ: large-scale distributed matrix computation for the cloud

Proceedings of the 7th ACM european conference on Computer Systems
A down-to-earth look at the cloud host OS

Proceedings of the 1st International Workshop on Hot Topics in Cloud Data Processing
TransMR: data-centric programming beyond data parallelism

HotCloud'11 Proceedings of the 3rd USENIX conference on Hot topics in cloud computing
iMapReduce: A Distributed Computing Framework for Iterative Computation

Journal of Grid Computing
Optimizing analytic data flows for multiple execution engines

SIGMOD '12 Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data
Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing

NSDI'12 Proceedings of the 9th USENIX conference on Networked Systems Design and Implementation
Camdoop: exploiting in-network aggregation for big data applications

NSDI'12 Proceedings of the 9th USENIX conference on Networked Systems Design and Implementation
Re-optimizing data-parallel computing

NSDI'12 Proceedings of the 9th USENIX conference on Networked Systems Design and Implementation
Optimizing data shuffling in data-parallel computation by understanding user-defined functions

NSDI'12 Proceedings of the 9th USENIX conference on Networked Systems Design and Implementation
Adaptive MapReduce using situation-aware mappers

Proceedings of the 15th International Conference on Extending Database Technology
Massively-parallel stream processing under QoS constraints with Nephele

Proceedings of the 21st international symposium on High-Performance Parallel and Distributed Computing
Composition of engineering web services with universal distributed data-flows framework based on ROA

Proceedings of the Third International Workshop on RESTful Design
From a calculus to an execution environment for stream processing

Proceedings of the 6th ACM International Conference on Distributed Event-Based Systems
The only constant is change: incorporating time-varying network reservations in data centers

Proceedings of the ACM SIGCOMM 2012 conference on Applications, technologies, architectures, and protocols for computer communication
The seven deadly sins of cloud computing research

HotCloud'12 Proceedings of the 4th USENIX conference on Hot Topics in Cloud Ccomputing
Using R for iterative and incremental processing

HotCloud'12 Proceedings of the 4th USENIX conference on Hot Topics in Cloud Ccomputing
Composable reliability for asynchronous systems

USENIX ATC'12 Proceedings of the 2012 USENIX conference on Annual Technical Conference
Oolong: asynchronous distributed applications made easy

Proceedings of the Asia-Pacific Workshop on Systems
Spinning fast iterative data flows

Proceedings of the VLDB Endowment
Optimization of analytic data flows for next generation business intelligence applications

TPCTC'11 Proceedings of the Third TPC Technology conference on Topics in Performance Evaluation, Measurement and Characterization
The only constant is change: incorporating time-varying network reservations in data centers

ACM SIGCOMM Computer Communication Review - Special october issue SIGCOMM '12
Oolong: asynchronous distributed applications made easy

APSys'12 Proceedings of the Third ACM SIGOPS Asia-Pacific conference on Systems
Turbine: a distributed-memory dataflow engine for extreme-scale many-task applications

Proceedings of the 1st ACM SIGMOD Workshop on Scalable Workflow Execution Engines and Technologies
Xen2MX: towards high-performance communication in the cloud

Euro-Par'12 Proceedings of the 18th international conference on Parallel processing workshops
Revisiting flow-based load balancing: Stateless path selection in data center networks

Computer Networks: The International Journal of Computer and Telecommunications Networking
Optimus: a dynamic rewriting framework for data-parallel execution plans

Proceedings of the 8th ACM European Conference on Computer Systems
Presto: distributed machine learning and graph processing with sparse matrices

Proceedings of the 8th ACM European Conference on Computer Systems
Large-scale computation not at the cost of expressiveness

HotOS'13 Proceedings of the 14th USENIX conference on Hot Topics in Operating Systems
New wine in old skins: the case for distributed operating systems in the data center

Proceedings of the 4th Asia-Pacific Workshop on Systems
Data-Intensive Cloud Computing: Requirements, Expectations, Challenges, and Solutions

Journal of Grid Computing
Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles

ACM SIGOPS 24th Symposium on Operating Systems Principles
Dandelion: a compiler and runtime for heterogeneous systems

Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles
Naiad: a timely dataflow system

Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles
Piranha: optimizing short jobs in Hadoop

Proceedings of the VLDB Endowment
MillWheel: fault-tolerant stream processing at internet scale

Proceedings of the VLDB Endowment
Hybrid Analytic Flows-the Case for Optimization

Fundamenta Informaticae - Scalable Workflow Enactment Engines and Technology
Turbine: A Distributed-memory Dataflow Engine for High Performance Many-task Applications

Fundamenta Informaticae - Scalable Workflow Enactment Engines and Technology
Nephele streaming: stream processing under QoS constraints at scale

Cluster Computing

Quantified Score

Hi-index	0.00

Visualization

Abstract

This paper introduces CIEL, a universal execution engine for distributed data-flow programs. Like previous execution engines, CIEL masks the complexity of distributed programming. Unlike those systems, a CIEL job can make data-dependent control-flow decisions, which enables it to compute iterative and recursive algorithms. We have also developed Skywriting, a Turing-complete scripting language that runs directly on CIEL. The execution engine provides transparent fault tolerance and distribution to Skywriting scripts and high-performance code written in other programming languages. We have deployed CIEL on a cloud computing platform, and demonstrate that it achieves scalable performance for both iterative and non-iterative algorithms.