Optimizing I/O forwarding techniques for extreme-scale event tracing

  • Authors:

  • Thomas Ilsche;Joseph Schuchart;Jason Cope;Dries Kimpe;Terry Jones;Andreas Knüpfer;Kamil Iskra;Robert Ross;Wolfgang E. Nagel;Stephen Poole

  • Affiliations:

  • Technische Universität Dresden (ZIH), Dresden, Germany 01062;Oak Ridge National Laboratory, Oak Ridge, USA 37831;Argonne National Laboratory, Argonne, USA 60439;Argonne National Laboratory, Argonne, USA 60439;Oak Ridge National Laboratory, Oak Ridge, USA 37831;Technische Universität Dresden (ZIH), Dresden, Germany 01062;Argonne National Laboratory, Argonne, USA 60439;Argonne National Laboratory, Argonne, USA 60439;Technische Universität Dresden (ZIH), Dresden, Germany 01062;Oak Ridge National Laboratory, Oak Ridge, USA 37831

  • Venue:
  • Cluster Computing
  • Year:
  • 2014

Quantified Score

Hi-index 0.00

Visualization

Abstract

Programming development tools are a vital component for understanding the behavior of parallel applications. Event tracing is a principal ingredient to these tools, but new and serious challenges place event tracing at risk on extreme-scale machines. As the quantity of captured events increases with concurrency, the additional data can overload the parallel file system and perturb the application being observed. In this work we present a solution for event tracing on extreme-scale machines. We enhance an I/O forwarding software layer to aggregate and reorganize log data prior to writing to the storage system, significantly reducing the burden on the underlying file system. Furthermore, we introduce a sophisticated write buffering capability to limit the impact. To validate the approach, we employ the Vampir tracing toolset using these new capabilities. Our results demonstrate that the approach increases the maximum traced application size by a factor of 5脳 to more than 200,000 processes.