Virtual memory, processes, and sharing in MULTICS
Communications of the ACM
Linkers and Loaders
Overcoming Scalability Challenges for Tool Daemon Launching
ICPP '08 Proceedings of the 2008 37th International Conference on Parallel Processing
Experiences with a Lightweight Supercomputer Kernel: Lessons Learned from Blue Gene's CNK
Proceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis
Benefits of high speed interconnects to cluster file systems: a case study with lustre
IPDPS'06 Proceedings of the 20th international conference on Parallel and distributed processing
A Peer-to-Peer Architecture for Supporting Dynamic Shared Libraries in Large-Scale Systems
ICPPW '12 Proceedings of the 2012 41st International Conference on Parallel Processing Workshops
Hi-index | 0.00 |
Dynamic linking has many advantages for managing large code bases, but dynamically linked applications have not typically scaled well on high performance computing systems. Splitting a monolithic executable into many dynamic shared object (DSO) files decreases compile time for large codes, reduces runtime memory requirements by allowing modules to be loaded and unloaded as needed, and allows common DSOs to be shared among many executables. However, launching an executable that depends on many DSOs causes a flood of file system operations at program start-up, when each process in the parallel application loads its dependencies. At large scales, this operation has an effect similar to a site-wide denial-of-service attack, as even large parallel file systems struggle to service so many simultaneous requests. In this paper, we present SPINDLE, a novel approach to parallel loading that coordinates simultaneous file system operations with a scalable network of cache server processes. Our approach is transparent to user applications. We extend the GNU loader, which is used in Linux as well as proprietary operating systems, to limit the number of simultaneous file system operations, quickly loading DSOs without thrashing the file system. Our experiments show that our prototype implementation has a low overhead and increases the scalability of Pynamic, a benchmark that stresses the dynamic loader, by a factor of 20.