Failure recovery: when the cure is worse than the disease

  • Authors:
  • Zhenyu Guo;Sean McDirmid;Mao Yang;Li Zhuang;Pu Zhang;Yingwei Luo;Tom Bergan;Peter Bodik;Madan Musuvathi;Zheng Zhang;Lidong Zhou

  • Affiliations:
  • Microsoft Research;Microsoft Research;Microsoft Research;Microsoft Research;Microsoft Research and Peking University;Peking University;Microsoft Research and University of Washington;Microsoft Research;Microsoft Research;Microsoft Research;Microsoft Research

  • Venue:
  • HotOS'13 Proceedings of the 14th USENIX conference on Hot Topics in Operating Systems
  • Year:
  • 2013

Quantified Score

Hi-index 0.00

Visualization

Abstract

Cloud services inevitably fail: machines lose power, networks become disconnected, pesky software bugs cause sporadic crashes, and so on. Unfortunately, failure recovery itself is often faulty; e.g. recovery can accidentally recursively replicate small failures to other machines until the entire cloud service fails in a catastrophic outage, amplifying a small cold into a contagious deadly plague! We propose that failure recovery should be engineered foremost according to the maxim of primum non nocere, that it "does no harm." Accordingly, we must consider the system holistically when failure occurs and recover only when observed activity safely allows for it.