SIGMOD '86 Proceedings of the 1986 ACM SIGMOD international conference on Management of data
Concurrent file operations in a high performance
Proceedings of the 1992 ACM/IEEE conference on Supercomputing
The high performance storage system
Proceedings of the 1993 ACM/IEEE conference on Supercomputing
Parallel file systems for the IBM SP computers
IBM Systems Journal
Serverless network file systems
SOSP '95 Proceedings of the fifteenth ACM symposium on Operating systems principles
The Vesta parallel file system
ACM Transactions on Computer Systems (TOCS)
The galley parallel file system
ICS '96 Proceedings of the 10th international conference on Supercomputing
Experimental evaluation of the Hewlett-Parkard exemplar file system
ACM SIGMETRICS Performance Evaluation Review
A case for using MPI's derived datatypes to improve I/O performance
SC '98 Proceedings of the 1998 ACM/IEEE conference on Supercomputing
Meta-data Management System for High-Performance Large-Scale Scientific Data Access
HiPC '00 Proceedings of the 7th International Conference on High Performance Computing
The SDSC storage resource broker
CASCON '98 Proceedings of the 1998 conference of the Centre for Advanced Studies on Collaborative research
Data Sieving and Collective I/O in ROMIO
FRONTIERS '99 Proceedings of the The 7th Symposium on the Frontiers of Massively Parallel Computation
Data Management for Large-Scale Scientific Computations in High Performance Distributed Systems
HPDC '99 Proceedings of the 8th IEEE International Symposium on High Performance Distributed Computing
HPDC '00 Proceedings of the 9th IEEE International Symposium on High Performance Distributed Computing
A distributed multi-storage I/O system for data intensive scientific computing
Parallel Computing - Special issue: Parallel and distributed scientific and engineering computing
PVFS: a parallel file system for linux clusters
ALS'00 Proceedings of the 4th annual Linux Showcase & Conference - Volume 4
A high-performance application data environment for large-scale scientific computations
IEEE Transactions on Parallel and Distributed Systems
Striping without sacrifices: maintaining POSIX semantics in a parallel file system
LASCO'08 First USENIX Workshop on Large-Scale Computing
| Hi-index | 0.00 |
One of the challenges brought by large-scale scientific applications is how to avoid remote storage access by collectively using sufficient local storage resources to hold huge amounts of data generated by the simulation while providing high-performance I/O. DPFS, a distributed parallel file system, is designed and implemented to address this problem. DPFS collects locally distributed and unused storage resources as a supplement to the internal storage of parallel computing systems to satisfy the storage capacity requirement of large-scale applications. In addition, like parallel file systems, DPFS provides striping mechanisms that divide a file into small pieces and distributes them across multiple storage devices for parallel data access. The unique feature of DPFS is that it provides three file levels with each file level corresponding to a file striping method. In addition to the traditional linear striping method, DPFS also provides a novel Multidimensional striping method that can solve performance problems of linear striping for many popular access patterns. Other issues such as load-balancing and user interface are also addressed in DPFS.