A Case Study into Using Common Real-Time Workflow Monitoring Infrastructure for Scientific Workflows

  • Authors:

  • Karan Vahi;Ian Harvey;Taghrid Samak;Daniel Gunter;Kieran Evans;David Rogers;Ian Taylor;Monte Goode;Fabio Silva;Eddie Al-Shakarchi;Gaurang Mehta;Ewa Deelman;Andrew Jones

  • Affiliations:

  • USC Information Sciences Institute, Marina Del Rey, USA;School of Computer Science, Cardiff, UK;Lawrence Berkeley National Laboratory, Berkeley, USA;Lawrence Berkeley National Laboratory, Berkeley, USA;School of Computer Science, Cardiff, UK;School of Computer Science, Cardiff, UK;School of Computer Science, Cardiff, UK;Lawrence Berkeley National Laboratory, Berkeley, USA;University of Southern California, Los Angeles, USA;School of Computer Science, Cardiff, UK;USC Information Sciences Institute, Marina Del Rey, USA;USC Information Sciences Institute, Marina Del Rey, USA;School of Computer Science, Cardiff, UK

  • Venue:
  • Journal of Grid Computing
  • Year:
  • 2013

Quantified Score

Hi-index 0.00

Visualization

Abstract

Scientific workflow systems support various workflow representations, operational modes, and configurations. Regardless of the system used, end users have common needs: to track the status of their workflows in real time, be notified of execution anomalies and failures automatically, perform troubleshooting, and automate the analysis of the workflow results. In this paper, we describe how the Stampede monitoring infrastructure was integrated with the Pegasus Workflow Management System and the Triana Workflow Systems, in order to add generic real time monitoring and troubleshooting capabilities across both systems. Stampede is an infrastructure that provides interoperable monitoring using a three-layer model: (1) a common data model to describe workflow and job executions; (2) high-performance tools to load workflow logs conforming to the data model into a data store; and (3) a common query interface. This paper describes the integration of Stampede monitoring architecture with Pegasus and Triana and shows the new analysis capabilities that Stampede provides to these workflow systems. The successful integration of Stampede with these workflow engines demonstrates the generic nature of the Stampede monitoring infrastructure and its potential to provide a common platform for monitoring across scientific workflow engines.