A fully automated fault-tolerant system for distributed video processing and off-site replication

Authors:
George Kola;Tevfik Kosar;Miron Livny
Affiliations:
University of Wisconsin-Madison, Madison WI;University of Wisconsin-Madison, Madison WI;University of Wisconsin-Madison, Madison WI
Venue:
NOSSDAV '04 Proceedings of the 14th international workshop on Network and operating systems support for digital audio and video
Year:
2004

Citing 3
Cited 7

DEVise: integrated querying and visual exploration of large datasets

SIGMOD '97 Proceedings of the 1997 ACM SIGMOD international conference on Management of data
Stork: Making Data Placement a First Class Citizen in the Grid

ICDCS '04 Proceedings of the 24th International Conference on Distributed Computing Systems (ICDCS'04)
Explicit control a batch-aware distributed file system

NSDI'04 Proceedings of the 1st conference on Symposium on Networked Systems Design and Implementation - Volume 1

Phoenix: Making Data-Intensive Grid Applications Fault-Tolerant

GRID '04 Proceedings of the 5th IEEE/ACM International Workshop on Grid Computing
Dynamic partner identification in mobile agent-based distributed job workflow execution

Journal of Parallel and Distributed Computing
Intelligent data staging with overlapped execution of grid applications

Future Generation Computer Systems
A grid-enabled workflow system for reservoir uncertainty analysis

CLADE '08 Proceedings of the 6th international workshop on Challenges of large applications in distributed environments
Multi-Replication with Intelligent Staging in Data-Intensive Grid Applications

GRID '06 Proceedings of the 7th IEEE/ACM International Conference on Grid Computing
Data transfer planning with tree placement for collaborative environments

Constraints
Faults in large distributed systems and what we can do about them

Euro-Par'05 Proceedings of the 11th international Euro-Par conference on Parallel Processing

Quantified Score

Hi-index	0.00

Visualization

Abstract

Different fields including biomedical-engineering, educational research and geology have an increasing need to process large amounts of video and make them electronically available at different locations. So far, this has been a failure-prone tedious operation with an operator needed to babysit the processing and off-site replication of processed video. In this work, we developed a fault-tolerant system that handles large scale processing and replication of digital video in a fully automated manner. The system is highly resilient and handles a variety of hardware, software and network failures making it possible to process videos using commodity clusters or grid resources. Finally, we discuss how the system is being used in educational research to process several hundred terabytes of video.