A Distributed Workflow Management System with Case Study of Real-life Scientific Applications on Grids

  • Authors:

  • Qishi Wu;Mengxia Zhu;Yi Gu;Patrick Brown;Xukang Lu;Wuyin Lin;Yangang Liu

  • Affiliations:

  • Department of Computer Science, University of Memphis, Memphis, USA 38152;Department of Computer Science, Southern Illinois University, Carbondale, USA 62901;Dept of Management, Marketing, Computer Science, and Information Systems, University of Tennessee at Martin, Martin, USA 38238;Department of Computer Science, Southern Illinois University, Carbondale, USA 62901;Department of Computer Science, University of Memphis, Memphis, USA 38152;Atmospheric Science Division, Brookhaven National Laboratory, Upton, USA 11793;Atmospheric Science Division, Brookhaven National Laboratory, Upton, USA 11793

  • Venue:
  • Journal of Grid Computing
  • Year:
  • 2012

Quantified Score

Hi-index 0.00

Visualization

Abstract

Next-generation scientific applications feature complex workflows comprised of many computing modules with intricate inter-module dependencies. Supporting such scientific workflows in wide-area networks especially Grids and optimizing their performance are crucial to the success of collaborative scientific discovery. We develop a Scientific Workflow Automation and Management Platform (SWAMP), which enables scientists to conveniently assemble, execute, monitor, control, and steer computing workflows in distributed environments via a unified web-based user interface. The SWAMP architecture is built entirely on a seamless composition of web services: the functionalities of its own are provided and its interactions with other tools or systems are enabled through web services for easy access over standard Internet protocols while being independent of different platforms and programming languages. SWAMP also incorporates a class of efficient workflow mapping schemes to achieve optimal end-to-end performance based on rigorous performance modeling and algorithm design. The performance superiority of SWAMP over existing workflow mapping schemes is justified by extensive simulations, and the system efficacy is illustrated by large-scale experiments on real-life scientific workflows for climate modeling through effective system implementation, deployment, and testing on the Open Science Grid.