Characterizing and profiling scientific workflows

Authors:
Gideon Juve;Ann Chervenak;Ewa Deelman;Shishir Bharathi;Gaurang Mehta;Karan Vahi
Affiliations:
-;-;-;-;-;-
Venue:
Future Generation Computer Systems
Year:
2013

Citing 29
Cited 1

Scientific workload characterization by loop-based analyses

ACM SIGMETRICS Performance Evaluation Review
A Repository Service for Grid Workflow Components

ICAS-ICNS '05 Proceedings of the Joint International Conference on Autonomic and Autonomous Systems and International Conference on Networking and Services
Task scheduling strategies for workflow-based applications in grids

CCGRID '05 Proceedings of the Fifth IEEE International Symposium on Cluster Computing and the Grid (CCGrid'05) - Volume 2 - Volume 02
Application-Level Resource Provisioning on the Grid

E-SCIENCE '06 Proceedings of the Second IEEE International Conference on e-Science and Grid Computing
Managing Large-Scale Workflow Execution from Resource Provisioning to Provenance Tracking: The CyberShake Example

E-SCIENCE '06 Proceedings of the Second IEEE International Conference on e-Science and Grid Computing
Pegasus: A framework for mapping complex scientific workflows onto distributed systems

Scientific Programming
SCALEA-G: A unified monitoring and performance analysis system for the grid

Scientific Programming - AxGrids 2004
Performance metrics and ontologies for Grid workflows

Future Generation Computer Systems
Designing the myExperiment Virtual Research Environment for the Social Sharing of Workflows

E-SCIENCE '07 Proceedings of the Third IEEE International Conference on e-Science and Grid Computing
A Lightweight Middleware Monitor for Distributed Scientific Workflows

CCGRID '08 Proceedings of the 2008 Eighth IEEE International Symposium on Cluster Computing and the Grid
The cost of doing science on the cloud: the Montage example

Proceedings of the 2008 ACM/IEEE conference on Supercomputing
Towards a general model of the multi-criteria workflow scheduling on the grid

Future Generation Computer Systems
Reducing Time-to-Solution Using Distributed High-Throughput Mega-Workflows - Experiences from SCEC CyberShake

ESCIENCE '08 Proceedings of the 2008 Fourth IEEE International Conference on eScience
A Distributed Algorithm for Determining the Provenance of Data

ESCIENCE '08 Proceedings of the 2008 Fourth IEEE International Conference on eScience
On the Use of Cloud Computing for Scientific Workflows

ESCIENCE '08 Proceedings of the 2008 Fourth IEEE International Conference on eScience
How are Real Grids Used? The Analysis of Four Grid Traces and Its Implications

GRID '06 Proceedings of the 7th IEEE/ACM International Conference on Grid Computing
Pipeline-centric provenance model

Proceedings of the 4th Workshop on Workflows in Support of Large-Scale Science
A Fresh Perspective on Developing and Executing DAG-Based Distributed Applications: A Case-Study of SAGA-Based Montage

E-SCIENCE '09 Proceedings of the 2009 Fifth IEEE International Conference on e-Science
ParaTrac: a fine-grained profiler for data-intensive workflows

Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing
Using virtual grids to simplify application scheduling

IPDPS'06 Proceedings of the 20th international conference on Parallel and distributed processing
Analysis of approaches for supporting the Open Provenance Model: A case study of the Trident workflow workbench

Future Generation Computer Systems
Online scheduling of workflow applications in grid environments

Future Generation Computer Systems
Workflows to open provenance graphs, round-trip

Future Generation Computer Systems
Provenance management in Swift

Future Generation Computer Systems
Cost optimized provisioning of elastic resources for application workflows

Future Generation Computer Systems
Metrics for heterogeneous scientific workflows: A case study of an earthquake science application

International Journal of High Performance Computing Applications
Auto-scaling to minimize cost and meet application deadlines in cloud workflows

Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis
Deadline prediction scheduling based on benefits

Future Generation Computer Systems
Deadline-constrained workflow scheduling algorithms for Infrastructure as a Service Clouds

Future Generation Computer Systems

Toward fine-grained online task characteristics estimation in scientific workflows

WORKS '13 Proceedings of the 8th Workshop on Workflows in Support of Large-Scale Science

Quantified Score

Hi-index	0.00

Visualization

Abstract

Researchers working on the planning, scheduling, and execution of scientific workflows need access to a wide variety of scientific workflows to evaluate the performance of their implementations. This paper provides a characterization of workflows from six diverse scientific applications, including astronomy, bioinformatics, earthquake science, and gravitational-wave physics. The characterization is based on novel workflow profiling tools that provide detailed information about the various computational tasks that are present in the workflow. This information includes I/O, memory and computational characteristics. Although the workflows are diverse, there is evidence that each workflow has a job type that consumes the most amount of runtime. The study also uncovered inefficiency in a workflow component implementation, where the component was re-reading the same data multiple times.