Windows Azure Storage: a highly available cloud storage service with strong consistency

Authors:
Brad Calder;Ju Wang;Aaron Ogus;Niranjan Nilakantan;Arild Skjolsvold;Sam McKelvie;Yikang Xu;Shashwat Srivastav;Jiesheng Wu;Huseyin Simitci;Jaidev Haridas;Chakravarthy Uddaraju;Hemal Khatri;Andrew Edwards;Vaman Bedekar;Shane Mainali;Rafay Abbasi;Arpit Agarwal;Mian Fahim ul Haq;Muhammad Ikram ul Haq;Deepali Bhardwaj;Sowmya Dayanand;Anitha Adusumilli;Marvin McNett;Sriram Sankaran;Kavitha Manivannan;Leonidas Rigas
Affiliations:
Microsoft;Microsoft;Microsoft;Microsoft;Microsoft;Microsoft;Microsoft;Microsoft;Microsoft;Microsoft;Microsoft;Microsoft;Microsoft;Microsoft;Microsoft;Microsoft;Microsoft;Microsoft;Microsoft;Microsoft;Microsoft;Microsoft;Microsoft;Microsoft;Microsoft;Microsoft;Microsoft
Venue:
SOSP '11 Proceedings of the Twenty-Third ACM Symposium on Operating Systems Principles
Year:
2011

Citing 20
Cited 55

Managing update conflicts in Bayou, a weakly connected replicated storage system

SOSP '95 Proceedings of the fifteenth ACM symposium on Operating systems principles
The log-structured merge-tree (LSM-tree)

Acta Informatica
DCD—disk caching disk: a new approach for boosting I/O performance

ISCA '96 Proceedings of the 23rd annual international symposium on Computer architecture
The dangers of replication and a solution

SIGMOD '96 Proceedings of the 1996 ACM SIGMOD international conference on Management of data
The part-time parliament

ACM Transactions on Computer Systems (TOCS)
On optimistic methods for concurrency control

ACM Transactions on Database Systems (TODS)
Towards robust distributed systems (abstract)

Proceedings of the nineteenth annual ACM symposium on Principles of distributed computing
New directions in traffic measurement and accounting

Proceedings of the 2002 conference on Applications, technologies, architectures, and protocols for computer communications
The Google file system

SOSP '03 Proceedings of the nineteenth ACM symposium on Operating systems principles
Profiling over Adaptive Ranges

Proceedings of the International Symposium on Code Generation and Optimization
Chain replication for supporting high throughput and availability

OSDI'04 Proceedings of the 6th conference on Symposium on Opearting Systems Design & Implementation - Volume 6
Bigtable: a distributed storage system for structured data

OSDI '06 Proceedings of the 7th USENIX Symposium on Operating Systems Design and Implementation - Volume 7
The Chubby lock service for loosely-coupled distributed systems

OSDI '06 Proceedings of the 7th USENIX Symposium on Operating Systems Design and Implementation - Volume 7
Dynamo: amazon's highly available key-value store

Proceedings of twenty-first ACM SIGOPS symposium on Operating systems principles
PNUTS: Yahoo!'s hosted data serving platform

Proceedings of the VLDB Endowment
GFS: Evolution on Fast-forward

Queue - File Systems
Cassandra: a decentralized structured storage system

ACM SIGOPS Operating Systems Review
Object storage on CRAQ: high-throughput chain replication for read-mostly workloads

USENIX'09 Proceedings of the 2009 conference on USENIX Annual technical conference
VL2: a scalable and flexible data center network

Communications of the ACM
SnapMirror®: file system based asynchronous mirroring for disaster recovery

FAST'02 Proceedings of the 1st USENIX conference on File and storage technologies

A performance evaluation of Azure and Nimbus clouds for scientific applications

Proceedings of the 2nd International Workshop on Cloud Computing Platforms
Rethinking erasure codes for cloud file systems: minimizing I/O for recovery and degraded reads

FAST'12 Proceedings of the 10th USENIX conference on File and Storage Technologies
Walnut: a unified cloud object store

SIGMOD '12 Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data
Consistency and fault tolerance for erasure-coded distributed storage systems

Proceedings of the fifth international workshop on Data-Intensive Distributed Computing Date
Mobius: unified messaging and data serving for mobile apps

Proceedings of the 10th international conference on Mobile systems, applications, and services
TomusBlobs: Towards Communication-Efficient Storage for MapReduce Applications in Azure

CCGRID '12 Proceedings of the 2012 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (ccgrid 2012)
Erasure coding in windows azure storage

USENIX ATC'12 Proceedings of the 2012 USENIX conference on Annual Technical Conference
Redundantly grouped cross-object coding for repairable storage

Proceedings of the Asia-Pacific Workshop on Systems
Serializability, not serial: concurrency control and availability in multi-datacenter datastores

Proceedings of the VLDB Endowment
Transaction log based application error recovery and point in-time query

Proceedings of the VLDB Endowment
Redundantly grouped cross-object coding for repairable storage

APSys'12 Proceedings of the Third ACM SIGOPS Asia-Pacific conference on Systems
Supporting research collaboration through bi-level file synchronization

Proceedings of the 17th ACM international conference on Supporting group work
Reducing Storage Overhead with Small Write Bottleneck Avoiding in Cloud RAID System

GRID '12 Proceedings of the 2012 ACM/IEEE 13th International Conference on Grid Computing
µLibCloud: Providing High Available and Uniform Accessing to Multiple Cloud Storages

GRID '12 Proceedings of the 2012 ACM/IEEE 13th International Conference on Grid Computing
Improving Bandwidth Efficiency for Consistent Multistream Storage

ACM Transactions on Storage (TOS)
Pyramid Codes: Flexible Schemes to Trade Space for Access Efficiency in Reliable Data Storage Systems

ACM Transactions on Storage (TOS)
Pyxis: an active replication approach for enhancing social media services

AMT'12 Proceedings of the 8th international conference on Active Media Technology
An overview of codes tailor-made for better repairability in networked distributed storage systems

ACM SIGACT News
Paragon: QoS-aware scheduling for heterogeneous datacenters

Proceedings of the eighteenth international conference on Architectural support for programming languages and operating systems
COSBench: cloud object storage benchmark

Proceedings of the 4th ACM/SPEC International Conference on Performance Engineering
In-network redundancy generation for opportunistic speedup of data backup

Future Generation Computer Systems
GCplace: geo-cloud based correlation aware data replica placement

Proceedings of the 28th Annual ACM Symposium on Applied Computing
Stronger semantics for low-latency geo-replicated storage

nsdi'13 Proceedings of the 10th USENIX conference on Networked Systems Design and Implementation
Rhea: automatic filtering for unstructured cloud storage

nsdi'13 Proceedings of the 10th USENIX conference on Networked Systems Design and Implementation
Robustness in the Salus scalable block store

nsdi'13 Proceedings of the 10th USENIX conference on Networked Systems Design and Implementation
Verification of data redundancy in cloud storage

Proceedings of the 2013 international workshop on Security in cloud computing
Leveraging endpoint flexibility in data-intensive clusters

Proceedings of the ACM SIGCOMM 2013 conference on SIGCOMM
BlobCR: Virtual disk based checkpoint-restart for HPC applications on IaaS clouds

Journal of Parallel and Distributed Computing
XORing elephants: novel erasure codes for big data

Proceedings of the VLDB Endowment
Failure recovery: when the cure is worse than the disease

HotOS'13 Proceedings of the 14th USENIX conference on Hot Topics in Operating Systems
Using dark fiber to displace diesel generators

HotOS'13 Proceedings of the 14th USENIX conference on Hot Topics in Operating Systems
Replicated data consistency explained through baseball

Communications of the ACM
Regenerating codes: a system perspective

ACM SIGOPS Operating Systems Review
Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles

ACM SIGOPS 24th Symposium on Operating Systems Principles
LibRe: a consistency protocol for modern storage systems

Proceedings of the 6th ACM India Computing Convention
Consistency-based service level agreements for cloud storage

Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles
Naiad: a timely dataflow system

Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles
Natjam: design and evaluation of eviction policies for supporting priorities and deadlines in mapreduce clusters

Proceedings of the 4th annual Symposium on Cloud Computing
Scale-up vs scale-out for Hadoop: time to rethink?

Proceedings of the 4th annual Symposium on Cloud Computing
Network-aware data caching and prefetching for cloud-hosted metadata retrieval

NDM '13 Proceedings of the Third International Workshop on Network-Aware Data Management
Per-packet load-balanced, low-latency routing for clos-based data center networks

Proceedings of the ninth ACM conference on Emerging networking experiments and technologies
Copysets: reducing the frequency of data loss in cloud storage

USENIX ATC'13 Proceedings of the 2013 USENIX conference on Annual Technical Conference
On the efficiency of durable state machine replication

USENIX ATC'13 Proceedings of the 2013 USENIX conference on Annual Technical Conference
Providing a data location assurance service for cloud storage environments

Journal of Mobile Multimedia
CORFU: A distributed shared log

ACM Transactions on Computer Systems (TOCS)
Scalable transactions across heterogeneous NoSQL key-value data stores

Proceedings of the VLDB Endowment
QoS-Aware scheduling in heterogeneous datacenters with paragon

ACM Transactions on Computer Systems (TOCS)
Sector-Disk (SD) Erasure Codes for Mixed Failure Modes in RAID Systems

ACM Transactions on Storage (TOS)
Virtual Machine Coscheduling: A Game Theoretic Approach

UCC '13 Proceedings of the 2013 IEEE/ACM 6th International Conference on Utility and Cloud Computing
SD codes: erasure codes designed for how storage systems really fail

FAST'13 Proceedings of the 11th USENIX conference on File and Storage Technologies
Gecko: contention-oblivious disk arrays for cloud storage

FAST'13 Proceedings of the 11th USENIX conference on File and Storage Technologies
Screaming fast Galois field arithmetic using intel SIMD instructions

FAST'13 Proceedings of the 11th USENIX conference on File and Storage Technologies
Strata: scalable high-performance storage on virtualized non-volatile memory

FAST'14 Proceedings of the 12th USENIX conference on File and Storage Technologies
Parity logging with reserved space: towards efficient updates and recovery in erasure-coded clustered storage

FAST'14 Proceedings of the 12th USENIX conference on File and Storage Technologies
Exalt: empowering researchers to evaluate large-scale storage systems

NSDI'14 Proceedings of the 11th USENIX Conference on Networked Systems Design and Implementation

Quantified Score

Hi-index	0.02

Visualization

Abstract

Windows Azure Storage (WAS) is a cloud storage system that provides customers the ability to store seemingly limitless amounts of data for any duration of time. WAS customers have access to their data from anywhere at any time and only pay for what they use and store. In WAS, data is stored durably using both local and geographic replication to facilitate disaster recovery. Currently, WAS storage comes in the form of Blobs (files), Tables (structured storage), and Queues (message delivery). In this paper, we describe the WAS architecture, global namespace, and data model, as well as its resource provisioning, load balancing, and replication systems.