The Telescience Portal for advanced tomography applications
Journal of Parallel and Distributed Computing - Special issue on computational grids
Kepler: An Extensible System for Design and Execution of Scientific Workflows
SSDBM '04 Proceedings of the 16th International Conference on Scientific and Statistical Database Management
Scheduling of scientific workflows in the ASKALON grid environment
ACM SIGMOD Record
VLE-WFBus: A Scientific Workflow Bus for Multi e-Science Domains
E-SCIENCE '06 Proceedings of the Second IEEE International Conference on e-Science and Grid Computing
Pegasus: A framework for mapping complex scientific workflows onto distributed systems
Scientific Programming
Designing the myExperiment Virtual Research Environment for the Social Sharing of Workflows
E-SCIENCE '07 Proceedings of the Third IEEE International Conference on e-Science and Grid Computing
International Journal of High Performance Computing Applications
Flexible and Efficient Workflow Deployment of Data-Intensive Applications On Grids With MOTEUR
International Journal of High Performance Computing Applications
The Trident Scientific Workflow Workbench
ESCIENCE '08 Proceedings of the 2008 Fourth IEEE International Conference on eScience
Workflows and e-Science: An overview of workflow system features and capabilities
Future Generation Computer Systems
Pipeline-centric provenance model
Proceedings of the 4th Workshop on Workflows in Support of Large-Scale Science
Web enabling desktop workflow applications
Proceedings of the 4th Workshop on Workflows in Support of Large-Scale Science
Brokering multi-grid workflows in the P-GRADE portal
Euro-Par'06 Proceedings of the CoreGRID 2006, UNICORE Summit 2006, Petascale Computational Biology and Bioinformatics conference on Parallel processing
Why Linked Data is Not Enough for Scientists
ESCIENCE '10 Proceedings of the 2010 IEEE Sixth International Conference on e-Science
P-GRADE portal family for grid infrastructures
Concurrency and Computation: Practice & Experience
Online Fault and Anomaly Detection for Large-Scale Scientific Workflows
HPCC '11 Proceedings of the 2011 IEEE International Conference on High Performance Computing and Communications
Object reuse and exchange for publishing and sharing workflows
Proceedings of the 6th workshop on Workflows in support of large-scale science
Exploring workflow interoperability tools for neuroimaging data analysis
Proceedings of the 6th workshop on Workflows in support of large-scale science
IWIR: a language enabling portability across grid workflow systems
Proceedings of the 6th workshop on Workflows in support of large-scale science
Online workflow management and performance analysis with stampede
Proceedings of the 7th International Conference on Network and Services Management
| Hi-index | 0.00 |
In this paper, we leverage the previous work on the SHIWA bundling format and expand on this specification in order to facilitate workflow execution within a multi-workflow environment. We introduce a scalable and robust execution pool environment that supports workflows consisting of sub-workflows built upon a multitude of different workflow engines and environments, and also provide a common workflow representation for seamless connectivity through serialization to workflow bundles. We also present a meta-workflow scenario based upon this system. Workflow bundles employ the lightweight Open Archives Initiative Object Reuse and Exchange (ORE) Web-based standard, to provide a common format for representing and sharing workflows and the associated metadata required for their execution. This generalized bundling approach is already available within five workflow engines and has proven a useful environment for inter-workflow experimentation. The execution pool facilitates federated access to multiple distributed computing infrastructures supported by the underlying workflow engines subscribed to the pool. Workflow bundles are exposed using the eXtensible Messaging and Presence Protocol (XMPP), which provides the necessary communication backbone to enable multiple workflow engine agents to asynchronously publish and subscribe to bundles in meta-workflow pipelines. We present experiments showing the scalability and robustness of the pool execution approach with results showing that overheads remain controlled for up to 150 workflow agents, and that agent failures have very limited impact. We then demonstrate the applicability of our architecture by describing how a Java-based music analysis workflow can be distributed within such a multi-workflow environment consisting of the Triana and MOTEUR workflow engines.