CineGrid Exchange: A workflow-based peta-scale distributed storage platform on a high-speed network

Authors:
Shaofeng Liu;Jurgen P. Schulze;Laurin Herr;Jeffrey D. Weekley;Bing Zhu;Natalie V. Osdol;Dana Plepys;Mike Wan
Affiliations:
Department of Computer Science and Engineering, University of California San Diego (UCSD), La Jolla, CA, USA and California Institute for Telecommunications and Information Technology (Calit2), UC ...;California Institute for Telecommunications and Information Technology (Calit2), UC San Diego, La Jolla, CA, USA;Pacific Interface, Inc., Oakland, CA, USA;MOVES Institute, Naval Postgraduate School, Monterey, CA, USA;Institute for Neural Computation, University of California San Diego, La Jolla, CA, USA;Pacific Interface, Inc., Oakland, CA, USA;Electronic Visualization Laboratory, Department of Computer Science, University of Illinois at Chicago, Chicago, IL, USA;Institute for Neural Computation, University of California San Diego, La Jolla, CA, USA
Venue:
Future Generation Computer Systems
Year:
2011

Citing 7
Cited 1

Reliable Blast UDP: Predictable High Performance Bulk Data Transfer

CLUSTER '02 Proceedings of the IEEE International Conference on Cluster Computing
RobuSTore: a distributed storage architecture with robust and high performance

Proceedings of the 2007 ACM/IEEE conference on Supercomputing
The StarCAVE, a third-generation CAVE and virtual reality OptIPortal

Future Generation Computer Systems
The OptIPortal, a scalable visualization, storage, and computing interface device for the OptiPuter

Future Generation Computer Systems
The Rails Toolkit - Enabling End-System Topology-Aware High End Computing

ESCIENCE '08 Proceedings of the 2008 Fourth IEEE International Conference on eScience
Synchronizing Parallel Data Streams via Cross-Stream Coding

NAS '09 Proceedings of the 2009 IEEE International Conference on Networking, Architecture, and Storage
iRODS Primer: integrated Rule-Oriented Data System

iRODS Primer: integrated Rule-Oriented Data System

In-network redundancy generation for opportunistic speedup of data backup

Future Generation Computer Systems

Quantified Score

Hi-index	0.01

Visualization

Abstract

The Academy of Motion Picture Arts and Sciences (AMPAS) report ''The Digital Dilemma'' describes the issues caused by the rapid increase of storage requirements for long-term preservation and access of high quality digital media content. As one of the research communities focusing on very high quality digital content, CineGrid addresses these issues by building a global-scale distributed storage platform suitable for handling high quality digital media, which we call CineGrid Exchange (CX). Today, the CX connects seven universities and research laboratories in five countries, managing 400TB of storage, of which 250TB are dedicated to CineGrid. All of these sites are interconnected through a 10 Gbps dedicated optical network. The CX distributed repository holds digital motion pictures at HD, 2K and 4K resolutions, digital still images and digital audio in various formats. The goals of the CX are: (1) providing a 10 Gbps interconnected distributed platform for the CineGrid community to study digital content related issues, e.g., digital archiving, the movie production process, and network transfer/streaming protocols; (2) building a tool with which people can securely store, easily share and transfer very high definition digital content worldwide for exhibition and real-time collaboration; (3) automating digital policies through middleware and metadata management. In this publication, we introduce the architecture of the CX, resources managed by the CX and the implementation of the first series of CX management policies using the iRODS programmable middleware. We evaluate the first phase of CX platform implementation. We show that the CX has the potential to be a reliable and scalable digital management system.