SOSP '03 Proceedings of the nineteenth ACM symposium on Operating systems principles
A five-year study of file-system metadata
ACM Transactions on Storage (TOS)
Bigtable: a distributed storage system for structured data
OSDI '06 Proceedings of the 7th symposium on Operating systems design and implementation
The Art of Capacity Planning: Scaling Web Resources
The Art of Capacity Planning: Scaling Web Resources
Using a market economy to provision compute resources across planet-wide clusters
IPDPS '09 Proceedings of the 2009 IEEE International Symposium on Parallel&Distributed Processing
Towards characterizing cloud backend workloads: insights from Google compute clusters
ACM SIGMETRICS Performance Evaluation Review
Cloud resource usage: extreme distributions invalidating traditional capacity planning models
Proceedings of the 2nd international workshop on Scientific cloud computing
ARIA: automatic resource inference and allocation for mapreduce environments
Proceedings of the 8th ACM international conference on Autonomic computing
Storage provisioning and allocation in a large cloud environment
Proceedings of the 2012 workshop on Management of big data systems
Janus: optimal flash provisioning for cloud storage workloads
USENIX ATC'13 Proceedings of the 2013 USENIX conference on Annual Technical Conference
Regression-based utilization prediction algorithms: an empirical investigation
CASCON '13 Proceedings of the 2013 Conference of the Center for Advanced Studies on Collaborative Research
| Hi-index | 0.00 |
Provisioning scarce resources among competing users and jobs remains one of the primary challenges of operating large-scale, distributed computing environments. Distributed storage systems, in particular, typically rely on hard operator-set quotas to control disk allocation and enforce isolation for space and I/O bandwidth among disparate users. However, users and operators are very poor at predicting future requirements and, as a result, tend to over-provision grossly. For three years, we collected detailed usage information for data stored in distributed filesystems in a large private cloud spanning dozens of clusters on multiple continents. Specifically, we measured the disk space usage, I/O rate, and age of stored data for thousands of different engineering users and teams. We find that although the individual time series often have non-stable usage trends, regional aggregations, user classification, and ensemble forecasting methods can be combined to provide a more accurate prediction of future use for the majority of users. We applied this methodology for the storage users in one geographic region and back-tested these techniques over the past three years to compare our forecasts against actual usage. We find that by classifying a small subset of users with unforecastable trend changes due to known product launches, we can generate three-month out forecasts with mean absolute errors of less than 12%. This compares favorably to the amount of allocated but unused quota that is generally wasted with manual operator-set quotas.