A storage-centric analysis of MapReduce workloads: File popularity, temporal locality and arrival patterns

Authors:
Cristina L. Abad;Nathan Roberts;Yi Lu;Roy H. Campbell
Affiliations:
University of Illinois at Urbana-Champaign, USA;Yahoo! Inc., USA;University of Illinois at Urbana-Champaign, USA;University of Illinois at Urbana-Champaign, USA
Venue:
IISWC '12 Proceedings of the 2012 IEEE International Symposium on Workload Characterization (IISWC)
Year:
2012

Citing 0
Cited 3

Metadata Traces and Workload Models for Evaluating Big Storage Systems

UCC '12 Proceedings of the 2012 IEEE/ACM Fifth International Conference on Utility and Cloud Computing
Investigating hybrid SSD FTL schemes for Hadoop workloads

Proceedings of the ACM International Conference on Computing Frontiers
Generating request streams on Big Data using clustered renewal processes

Performance Evaluation

Quantified Score

Hi-index	0.00

Visualization

Abstract

A huge increase in data storage and processing requirements has lead to Big Data, for which next generation storage systems are being designed and implemented. However, we have a limited understanding of the workloads of Big Data storage systems. We consider the case of one common type of Big Data storage cluster: a cluster dedicated to supporting a mix of MapReduce jobs. We analyze 6-month traces from two large Hadoop clusters at Yahoo! and characterize the file popularity, temporal locality, and arrival patterns of the workloads. We identify several interesting properties and compare them with previous observations from web and media server workloads. To the best of our knowledge, this is the first study of how MapReduce workloads interact with the storage layer.