Tracking provenance in a virtual data grid

  • Authors:

  • Ben Clifford;Ian Foster;Jens-S. Voeckler;Michael Wilde;Yong Zhao

  • Affiliations:

  • Computation Institute, University of Chicago, Chicago, IL 60637, U.S.A.;Computation Institute, University of Chicago, Chicago, IL 60637, U.S.A. and Math & Computer Science Division, Argonne National Laboratory, Argonne, IL 60439, U.S.A.;USC Information Sciences Institute, Marina Del Rey, CA, U.S.A.;Computation Institute, University of Chicago, Chicago, IL 60637, U.S.A. and Math & Computer Science Division, Argonne National Laboratory, Argonne, IL 60439, U.S.A.;Computation Institute, University of Chicago, Chicago, IL 60637, U.S.A.

  • Venue:
  • Concurrency and Computation: Practice & Experience - The First Provenance Challenge
  • Year:
  • 2008

Quantified Score

Hi-index 0.00

Visualization

Abstract

The virtual data model allows data sets to be described prior to, and separately from, their physical materialization. We have implemented this model in a Virtual Data Language (VDL) and associated supporting tools, which provide for both the storage, query, and retrieval of virtual data set descriptions, and the automated, on-demand materialization of virtual data sets. We use a standardized data provenance challenge exercise to illustrate the powerful queries that can be performed on the data maintained by these tools, which for a single virtual data set can include three elements: the computational procedure(s) that must be executed to materialize the data set, the runtime log(s) produced by the execution of the computation(s), and optional metadata annotation(s) that associate application semantics with data and procedures. Copyright © 2007 John Wiley & Sons, Ltd.