Future Generation Computer Systems
Distributed Management of Massive Data: An Efficient Fine-Grain Data Access Scheme
High Performance Computing for Computational Science - VECPAR 2008
| Hi-index | 0.00 |
Large-scale parallel applications often produce immense quantities of data that need to be analyzed. To avoid performing repeated, costly disk accesses, analysis of large data sets generally requires a commensurately large amount of memory. While some data-analysis tools can easily be parallelized to distribute memory across a cluster, other tools are either difficult to parallelize or, in the case of simple data-analysis scripts with short lifespans, not worth the effort to parallelize. In this work, we present and analyze the performance of JumboMem, a simple, entirely user-level parallel program that enables unmodified sequential applications to access all of the memory in a cluster. Although there are many implementations of memory servers, all require either administrative privileges or program modifications. More importantly, no existing memory server has been evaluated on modern workstation clusters with high-speed networks, many nodes, and significant quantities of memory. This paper represents the first study of memory-server performance at supercomputing scales.