Flexibility and performance of parallel file systems
ACM SIGOPS Operating Systems Review
Performance of the gallery parallel file system
Proceedings of the fourth workshop on I/O in parallel and distributed systems: part of the federated computing research conference
The galley parallel file system
ICS '96 Proceedings of the 10th international conference on Supercomputing
Design and implementation of a parallel I/O runtime system for irregular applications
Journal of Parallel and Distributed Computing
Enhancing Data Migration Performance via Parallel Data Compression
IPDPS '02 Proceedings of the 16th International Parallel and Distributed Processing Symposium
Persistent Array Access Using Server-Directed I/O
SSDBM '96 Proceedings of the Eighth International Conference on Scientific and Statistical Database Management
Optimizing I/O for Irregular Applications on Distributed-Memory Machines
ParNum '99 Proceedings of the 4th International ACPC Conference Including Special Tracks on Parallel Numerics and Parallel Computing in Image Processing, Video Processing, and Multimedia: Parallel Computation
Disk Resident Arrays: An Array-Oriented I/O Library for Out-Of-Core Computations
FRONTIERS '96 Proceedings of the 6th Symposium on the Frontiers of Massively Parallel Computation
| Hi-index | 0.00 |
Poor parallel i/o performance has recently been recognized as a roadblock to scalability of parallel architectures, algorithms, and data sets. For i/o of large arrays, the storage of arrays by subarray divisions-chunking-has been shown to improve i/o performance substantially in many circumstances, In this paper we show how to increase the performance advantages of chunking by combining it with data compression, and describe the results of experiments with compressed chunks from scientific data sets on the Intel iPSC/860. For a particular fixed array size and compression ratio, uncompressed chunk i/o is faster than compressed chunk i/o when the number of processors is small; the reverse holds when the number of processors is large, as the cost of compression as spread over a larger number of processors. With good compression ratios and large numbers of processors, we obtained an effective logical i/o rate for compressed chunks that exceeds the theoretical possible maximum for uncompressed data, by adding compression to an existing chunked i/o library. Our results suggest that compression may be a good technique for handling sparse arrays in parallel i/o.