Achieving high performance on extremely large parallel machines: performance prediction and load balancing

• Authors: Laxmikant V. Kale;Gengbin Zheng
• Venue: Achieving high performance on extremely large parallel machines: performance prediction and load balancing
• Year: 2005

Parallel machines with an extremely large number of processors are now in operation. For example, the IBM BlueGene/L machine with 128K processors is currently being deployed. It is going to be a significant challenge for application developers to write parallel programs in order to exploit the enormous compute power available and manually scale their applications on such machines. Solving these problems involves finding suitable parallel programming models for such machines and addressing issues like load imbalance. This thesis explores processor virtualization in Charm++ programming model and employing migratable objects for programming petaflops class machines supported by parallel emulation for algorithm validation, parallel simulation for performance prediction, and using new kinds of automatic load balancing strategies to substantially address many of these challenges for programming very large machines. It is important to understand the performance of parallel applications on very large parallel machines. This thesis explores Parallel Discrete Event Simulation techniques to simulate parallel applications and predict their performance. We present a novel optimistic synchronization protocol which exploits the inherent determinacy in parallel applications to effectively reduce the synchronization overhead. Load balance problem presents significant challenges to applications to achieve scalability on very large machines. We study load balancing techniques and develop a spectrum of load balancing strategies motivated by several real-world applications. We optimize our load balancing strategies in multiple dimensions of criteria such as communication-aware load balancing, sub-step load balancing, and computation phase-aware load balancing. We have successfully scaled NAMD (a classical molecular dynamics application) to 1TF of peak performance on 3000 processors of PSC Lemieux, using the load balancing framework presented in this thesis. We further motivate the need for next generation load balancing strategies for petaflops class machines. We explore a novel design of a scalable hierarchical load balancing scheme, which incorporates an explicit memory cost control function to make it easy to adapt to extremely large machines with small memory footprint. This hierarchical load balancing scheme builds load data from instrumenting an application automatically at run-time on both computation and communication pattern. The load balancing strategy takes application communication pattern into account explicitly.