Time-bound analytic tasks on large datasets through dynamic configuration of workflows

Authors:
Yolanda Gil;Varun Ratnakar;Rishi Verma;Andrew Hart;Paul Ramirez;Chris Mattmann;Arni Sumarlidason;Samuel L. Park
Affiliations:
University of Southern California, Marina del Rey, CA;University of Southern California, Marina del Rey, CA;NASA Jet Propulsion Laboratory, Pasadena, CA;NASA Jet Propulsion Laboratory, Pasadena, CA;NASA Jet Propulsion Laboratory, Pasadena, CA;NASA Jet Propulsion Laboratory, Pasadena, CA;MDA Information Systems LLC, Gaithersburg, MD;MDA Information Systems LLC, Gaithersburg, MD
Venue:
WORKS '13 Proceedings of the 8th Workshop on Workflows in Support of Large-Scale Science
Year:
2013

Citing 18
Cited 0

Latent dirichlet allocation

The Journal of Machine Learning Research
How Well Can Simple Metrics Represent the Performance of HPC Applications?

SC '05 Proceedings of the 2005 ACM/IEEE conference on Supercomputing
A software architecture-based framework for highly distributed and data intensive scientific applications

Proceedings of the 28th international conference on Software engineering
Examining the Challenges of Scientific Workflows

Computer
Scientific Software as Workflows: From Discovery to Distribution

IEEE Software
The design and realisation of the Experimentmy Virtual Research Environment for social sharing of workflows

Future Generation Computer Systems
PLDA: Parallel Latent Dirichlet Allocation for Large-Scale Applications

AAIM '09 Proceedings of the 5th International Conference on Algorithmic Aspects in Information and Management
A Reusable Process Control System Framework for the Orbiting Carbon Observatory and NPP Sounder PEATE Missions

SMC-IT '09 Proceedings of the Third IEEE International Conference on Space Mission Challenges for Information Technology
Expressive Reusable Workflow Templates

E-SCIENCE '09 Proceedings of the 2009 Fifth IEEE International Conference on e-Science
Parameterized specification, configuration and execution of data-intensive scientific workflows

Cluster Computing
Wings: Intelligent Workflow-Based Design of Computational Experiments

IEEE Intelligent Systems
A semantic framework for automatic generation of computational workflows using distributed data and component catalogues

Journal of Experimental & Theoretical Artificial Intelligence
A new approach for publishing workflows: abstractions, standards, and linked data

Proceedings of the 6th workshop on Workflows in support of large-scale science
Making data analysis expertise broadly accessible through workflows

Proceedings of the 6th workshop on Workflows in support of large-scale science
A Framework for Efficient Data Analytics through Automatic Configuration and Customization of Scientific Workflows

ESCIENCE '11 Proceedings of the 2011 IEEE Seventh International Conference on eScience
Tika in Action

Tika in Action
Predicting the Execution Time of Workflow Activities Based on Their Input Features

SCC '12 Proceedings of the 2012 SC Companion: High Performance Computing, Networking Storage and Analysis
A General Approach to Real-Time Workflow Monitoring

SCC '12 Proceedings of the 2012 SC Companion: High Performance Computing, Networking Storage and Analysis

Quantified Score

Hi-index	0.00

Visualization

Abstract

Domain experts are often untrained in big data technologies and this limits their ability to exploit the data they have available. Workflow systems hide the complexities of high-end computing and software engineering by offering pre-packaged analytic steps combined into multi-step methods commonly used by experts. A current limitation of workflow systems is that they do not take into account user deadlines: they run workflows selected by the user, but take their time to do so. This is impractical when large datasets are at stake, since users often prefer to see an answer faster even if it has lower precision or quality. In this paper, we present an extension to workflow systems that enables them to take into account user deadlines by automatically generating alternative workflow candidates and ranking them according to performance estimates. The system makes these estimates based on workflow performance models created from workflow executions, and uses semantic technologies to reason about workflow options. Possible workflow candidates are presented to the user in a compact manner, and are ranked according to their runtime estimates. We have implemented this approach in the WOOT system, which combines and extends capabilities from the WINGS semantic workflow system and the Apache OODT Object Oriented Data Technology and workflow execution system.