MadLINQ: large-scale distributed matrix computation for the cloud

  • Authors:

  • Zhengping Qian;Xiuwei Chen;Nanxi Kang;Mingcheng Chen;Yuan Yu;Thomas Moscibroda;Zheng Zhang

  • Affiliations:

  • Microsoft Research Asia, Beijing, China;Microsoft Research Asia, Beijing, China;Shanghai Jiaotong University, Shanghai, China;Shanghai Jiaotong University, Shanghai, China;Microsoft Research Silicon Valley, Mountain View, USA;Microsoft Research Asia, Beijing, China;Microsoft Research Asia, Beijing, China

  • Venue:
  • Proceedings of the 7th ACM european conference on Computer Systems
  • Year:
  • 2012

Quantified Score

Hi-index 0.00

Visualization

Abstract

The computation core of many data-intensive applications can be best expressed as matrix computations. The MadLINQ project addresses the following two important research problems: the need for a highly scalable, efficient and fault-tolerant matrix computation system that is also easy to program, and the seamless integration of such specialized execution engines in a general purpose data-parallel computing system. MadLINQ exposes a unified programming model to both matrix algorithm and application developers. Matrix algorithms are expressed as sequential programs operating on tiles (i.e., sub-matrices). For application developers, MadLINQ provides a distributed matrix computation library for .NET languages. Via the LINQ technology, MadLINQ also seamlessly integrates with DryadLINQ, a data-parallel computing system focusing on relational algebra. The system automatically handles the parallelization and distributed execution of programs on a large cluster. It outperforms current state-of-the-art systems by employing two key techniques, both of which are enabled by the matrix abstraction: exploiting extra parallelism using fine-grained pipelining and efficient on-demand failure recovery using a distributed fault-tolerant execution engine. We describe the design and implementation of MadLINQ and evaluate system performance using several real-world applications.