Lineage retrieval for scientific data processing: a survey
ACM Computing Surveys (CSUR)
A taxonomy of scientific workflow systems for grid computing
ACM SIGMOD Record
Introduction to scientific workflow management and the Kepler system
Proceedings of the 2006 ACM/IEEE conference on Supercomputing
Reaching for the Stars with Python
Computing in Science and Engineering
Querying and Creating Visualizations by Analogy
IEEE Transactions on Visualization and Computer Graphics
The Sloan Digital Sky Survey and beyond
ACM SIGMOD Record - Tribute to honor Jim Gray
Provenance for Computational Tasks: A Survey
Computing in Science and Engineering
Provenance in Comparative Analysis: A Study in Cosmology
Computing in Science and Engineering
Provenance and Annotation of Data and Processes: Second International Provenance and Annotation Workshop, IPAW 2008, Salt Lake City, UT, USA, June 17-18, 2008. Revised Selected Papers
Using Provenance to Support Real-Time Collaborative Design of Workflows
Provenance and Annotation of Data and Processes
C-SWF: A Lightweight Scientific Workflow System for Astronomical Data Processing
IWCSE '09 Proceedings of the 2009 Second International Workshop on Computer Science and Engineering - Volume 02
Astro-WISE: Tracing and Using Lineage for Scientific Data Processing
NBIS '09 Proceedings of the 2009 International Conference on Network-Based Information Systems
Provenance-based validation of e-science experiments
ISWC'05 Proceedings of the 4th international conference on The Semantic Web
Hi-index | 0.00 |
An important challenge facing e-Science is the development of scalable systems and analysis techniques that allow client applications to locate data and services in increasingly large-scale distributed environments. e-Science Systems should achieve three main goals: (i) efficient and selective processing of data, (ii) support network collaboration without clogging distribution networks; and (iii) allow transparency of experiments through repeatability and verifiability of experiments. Several systems have addressed limited combinations of these properties, but we address all three in this work. We describe the architecture and implementation of such a framework in Astro-WISE, an astronomical approach to distributed data processing, discovery and retrieval of datasets that achieves scalability via dynamic linking (data lineage) maintained within the system. We show that lineage data collected during the processing and analysis of datasets can be reused to perform selective reprocessing(at sub-image level)ondatasets while the remainder of the dataset is untouched, a rather difficult process to automate without lineage.