GridFlow: Workflow Management for Grid Computing
CCGRID '03 Proceedings of the 3st International Symposium on Cluster Computing and the Grid
The Grid 2: Blueprint for a New Computing Infrastructure
The Grid 2: Blueprint for a New Computing Infrastructure
Scientific workflow management and the Kepler system: Research Articles
Concurrency and Computation: Practice & Experience - Workflow in Grid Systems
Supernova Recognition Using Support Vector Machines
ICMLA '06 Proceedings of the 5th International Conference on Machine Learning and Applications
Provenance for Visualizations: Reproducibility and Beyond
Computing in Science and Engineering
Querying and re-using workflows with VsTrails
Proceedings of the 2008 ACM SIGMOD international conference on Management of data
A study on a lightweight scientific workflow system for astronomical e-science service
IITA'09 Proceedings of the 3rd international conference on Intelligent information technology application
| Hi-index | 0.00 |
Observational astrophysics has recently become a data-intensive science after many decades of relative data poverty. As a result, many of the algorithms developed for processing astronomical data, although well established for low-volume data capture, do not scale well to today's high-volume sky surveys and transient searches. Specifically, problems may occur with data transfer, workflow management, efficient parallelization, and integration of legacy code. Observational astrophysics workflows present computational challenges unique in high performance computing, including 24/7 operations, time-critical processing, and very large numbers of relatively small data files which must all be processed and archived. We present a case study based on Sunfall, a distributed, parallel scientific workflow system we built for the Nearby Supernova Factory, the largest data-volume supernova search currently in existence. We describe innovative techniques for data transfer and workflow management, and discuss lessons learned in building a large-scale observational astrophysics workflow management system.