Detection and correction of silent data corruption for large-scale high-performance computing

  • Authors:

  • David Fiala;Frank Mueller;Christian Engelmann;Rolf Riesen;Kurt Ferreira;Ron Brightwell

  • Affiliations:

  • North Carolina State University, Raleigh, NC;North Carolina State University, Raleigh, NC;Oak Ridge Natl Lab, Oak Ridge, TN;IBM Ireland, Dublin, Ireland;Sandia Natl Labs, Albuquerque, NM;Sandia Natl Labs, Albuquerque, NM

  • Venue:
  • SC '12 Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis
  • Year:
  • 2012

Quantified Score

Hi-index 0.00

Visualization

Abstract

Faults have become the norm rather than the exception for high-end computing clusters. Exacerbating this situation, some of these faults remain undetected, manifesting themselves as silent errors that allow applications to compute incorrect results. This paper studies the potential for redundancy to detect and correct soft errors in MPI message-passing applications while investigating the challenges inherent to detecting soft errors within MPI applications by providing transparent MPI redundancy. By assuming a model wherein corruption in application data manifests itself by producing differing MPI messages between replicas, we study the best suited protocols for detecting and correcting corrupted MPI messages. Using our fault injector, we observe that even a single error can have profound effects on applications by causing a cascading pattern of corruption which in most cases spreads to all other processes. Results indicate that our consistency protocols can successfully protect applications experiencing even high rates of silent data corruption.