MapReduce-based dimensional ETL made easy

Authors:
Xiufeng Liu;Christian Thomsen;Torben Bach Pedersen
Affiliations:
Aalborg University, Denmark;Aalborg University, Denmark;Aalborg University, Denmark
Venue:
Proceedings of the VLDB Endowment
Year:
2012

Citing 4
Cited 2

MapReduce: simplified data processing on large clusters

OSDI'04 Proceedings of the 6th conference on Symposium on Opearting Systems Design & Implementation - Volume 6
MapReduce and parallel DBMSs: friends or foes?

Communications of the ACM - Amir Pnueli: Ahead of His Time
pygrametl: a powerful programming framework for extract-transform-load programmers

Proceedings of the ACM twelfth international workshop on Data warehousing and OLAP
ETLMR: a highly scalable dimensional ETL framework based on mapreduce

DaWaK'11 Proceedings of the 13th international conference on Data warehousing and knowledge discovery

Daisy: the center for data-intensive systems at Aalborg University

ACM SIGMOD Record
Towards a conceptualization of ETL and physical storage of semantic data warehouses as a service

Cluster Computing

Quantified Score

Hi-index	0.00

Visualization

Abstract

This paper demonstrates ETLMR, a novel dimensional Extract--Transform--Load (ETL) programming framework that uses Map-Reduce to achieve scalability. ETLMR has built-in native support of data warehouse (DW) specific constructs such as star schemas, snowflake schemas, and slowly changing dimensions (SCDs). This makes it possible to build MapReduce-based dimensional ETL flows very easily. The ETL process can be configured with only few lines of code. We will demonstrate the concrete steps in using ETLMR to load data into a (partly snowflaked) DW schema. This includes configuration of data sources and targets, dimension processing schemes, fact processing, and deployment. In addition, we also present the scalability on large data sets.