(Big)data in a virtualized world: volume, velocity, and variety in cloud datacenters

  • Authors:

  • Robert Birke;Mathias Björkqvist;Lydia Y. Chen;Evgenia Smirni;Ton Engbersen

  • Affiliations:

  • IBM Research Zurich Lab;IBM Research Zurich Lab;IBM Research Zurich Lab;College of William and Mary;IBM Research Zurich Lab

  • Venue:
  • FAST'14 Proceedings of the 12th USENIX conference on File and Storage Technologies
  • Year:
  • 2014

Quantified Score

Hi-index 0.00

Visualization

Abstract

Virtualization is the ubiquitous way to provide computation and storage services to datacenter end-users. Guaranteeing sufficient data storage and efficient data access is central to all datacenter operations, yet little is known of the effects of virtualization on storage workloads. In this study, we collect and analyze field data from production datacenters that operate within the private cloud paradigm, during a period of three years. The datacenters of our study consist of 8,000 physical boxes, hosting over 90,000 VMs, which in turn use over 22 PB of storage. Storage data is analyzed from the perspectives of volume, velocity, and variety of storage demands on virtual machines and of their dependency on other resources. In addition to the growth rate and churn rate of allocated and used storage volume, the trace data illustrates the impact of virtualization and consolidation on the velocity of IO reads and writes, including IO deduplication ratios and peak load analysis of co-located VMs. We focus on a variety of applications which are roughly classified as app, web, database, file, mail, and print, and correlate their storage and IO demands with CPU, memory, and network usage. This study provides critical storage workload characterization by showing usage trends and how application types create storage traffic in large datacenters.