Semantic-based distributed i/o with the paramedic framework

Authors:
Pavan Balaji;Wuchun Feng;Heshan Lin
Affiliations:
Argonne National Laboratory, Argonne, IL, USA;Virginia Tech, Blacksburg, VA, USA;North Carolina State University, Raleigh, NC, USA
Venue:
HPDC '08 Proceedings of the 17th international symposium on High performance distributed computing
Year:
2008

Citing 7
Cited 1

Remote I/O: fast access to distant storage

Proceedings of the fifth workshop on I/O in parallel and distributed systems
Data Sieving and Collective I/O in ROMIO

FRONTIERS '99 Proceedings of the The 7th Symposium on the Frontiers of Massively Parallel Computation
A parallel I/O mechanism for distributed systems

CLUSTER '04 Proceedings of the 2004 IEEE International Conference on Cluster Computing
RFS: efficient and flexible remote file access for MPI-IO

CLUSTER '04 Proceedings of the 2004 IEEE International Conference on Cluster Computing
Parallel genomic sequence-searching on an ad-hoc grid: experiences, lessons learned, and implications

Proceedings of the 2006 ACM/IEEE conference on Supercomputing
MapReduce: simplified data processing on large clusters

OSDI'04 Proceedings of the 6th conference on Symposium on Opearting Systems Design & Implementation - Volume 6
MPI-IO/L: efficient remote I/O for MPI-IO via logistical networking

IPDPS'06 Proceedings of the 20th international conference on Parallel and distributed processing

Investigation into scaling I/O bound streaming applications productively with an all-FPGA cluster

Parallel Computing

Quantified Score

Hi-index	0.00

Visualization

Abstract

Many large-scale applications simultaneously rely on multiple resources for efficient execution. For example, such applications may require both large compute and storage resources; however, very few supercomputing centers can provide large quantities of both. Thus, data generated at the compute site oftentimes has to be moved to a remote storage site for either storage or visualization and analysis. Clearly, this is not an efficient model, especially when the two sites are distributed over a wide-area network. Thus, we present a framework called "ParaMEDIC: Parallel Metadata Environment for Distributed I/O and Computing" which uses application-specific semantic information to convert the generated data to orders-of-magnitude smaller metadata at the compute site, transfer the metadata to the storage site, and re-process the metadata at the storage site to regenerate the output. Specifically, ParaMEDIC trades a small amount of additional computation (in the form of data post-processing) for a potentially significant reduction in data that needs to be transferred in distributed environments.