Why do internet services fail, and what can be done about it?

  • Authors:

  • David Oppenheimer;Archana Ganapathi;David A. Patterson

  • Affiliations:

  • University of California at Berkeley, EECS Computer Science Division, Berkeley, CA;University of California at Berkeley, EECS Computer Science Division, Berkeley, CA;University of California at Berkeley, EECS Computer Science Division, Berkeley, CA

  • Venue:
  • USITS'03 Proceedings of the 4th conference on USENIX Symposium on Internet Technologies and Systems - Volume 4
  • Year:
  • 2003

Quantified Score

Hi-index 0.00

Visualization

Abstract

In 1986 Jim Gray published his landmark study of the causes of failures of Tandem systems and the techniques Tandem used to prevent such failures [6]. Seventeen years later, Internet services have replaced fault-tolerant servers as the new kid on the 24×7-availability block. Using data from three large-scale Internet services, we analyzed the causes of their failures and the (potential) effectiveness of various techniques for preventing and mitigating service failure. We find that (1) operator error is the largest cause of failures in two of the three services, (2) operator errors is the largest contributor to time to repair in two of the three services, (3) configuration errors are the largest category of operator errors, (4) failures in custom-written front-end software are significant, and (5) more extensive online testing and more thoroughly exposing and detecting component failures would reduce failure rates in at least one service. Qualitatively we find that improvement in the maintenance tools and systems used by service operations staff would decrease time to diagnose and repair problems.