Breaking the memory wall in MonetDB

  • Authors:
  • Peter A. Boncz;Martin L. Kersten;Stefan Manegold

  • Affiliations:
  • CWI, Kruislaan, Amsterdam, the Netherlands;CWI, Kruislaan, Amsterdam, the Netherlands;CWI, Kruislaan, Amsterdam, the Netherlands

  • Venue:
  • Communications of the ACM - Surviving the data deluge
  • Year:
  • 2008

Quantified Score

Hi-index 0.00

Visualization

Abstract

In the past decades, advances in speed of commodity CPUs have far outpaced advances in RAM latency. Main-memory access has therefore become a performance bottleneck for many computer applications; a phenomenon that is widely known as the "memory wall." In this paper, we report how research around the MonetDB database system has led to a redesign of database architecture in order to take advantage of modern hardware, and in particular to avoid hitting the memory wall. This encompasses (i) a redesign of the query execution model to better exploit pipelined CPU architectures and CPU instruction caches; (ii) the use of columnar rather than row-wise data storage to better exploit CPU data caches; (iii) the design of new cache-conscious query processing algorithms; and (iv) the design and automatic calibration of memory cost models to choose and tune these cache-conscious algorithms in the query optimizer.