Techniques for Optimizing Applications: High Performance Computing

• Authors: Rajat P. Garg;Ilya Sharapov
• Venue: Techniques for Optimizing Applications: High Performance Computing
• Year: 2002

From the Book:This book is a practical guide to optimizing performance of computationally intensive applications on Sun UltraSPARC platforms. It offers techniques for improving performance of applications that are predominantly compute-intensive or CPU-bound.We wrote this book with a general enough scope so that it would be useful to as many developers of technical applications on Sun platforms as possible. Also, we made the material practical by showing developers how to use each optimization method.For information on related topics such as system configuration and tuning, or improving the I/O and network performance, we refer readers to other resources.This book differs from other books and technical documents written aboutperformance optimization of high performance computing (HPC) applications. In many cases, other resources either give a detailed description of a product or provide general recommendations that are sometimes difficult to apply to practical tasks. In addition, some older resources are not as useful because of changes in technology.Though many of the techniques we offer apply to other platforms, we limited the scope of this guide to Sun compilers and UltraSPARC-based Solaris systems. We address new features in Sun compilers and in the Solaris Operating Environment, and we show readers how to use these products to get maximum performance on Sun hardware.Who Should Read This BookThis guide is primarily for developers of technical or HPC applications for Solaris. This audience includes both independent software vendor (ISV) developers and non-commercial developers.Developers creating or optimizing applications in the following fields may benefit from reading this book:Mechanical computer-aided engineering (MCAE)Electronic design automation (EDA)Computational chemistryBioinformaticsOperations researchFinancial modelingReservoir simulation and seismic modelingMechanical computer-aided design (MCAD) modelingGraphics rendering and imagingClimate and weather modelingThis book may also be helpful to technical application end-users in understanding the principles of HPC and how an application utilizes system resources.We assume the reader has:familiarity with development basics in UNIX environmentsa working knowledge of programming in C and Fortran languagesfamiliarity with computer architectureexperience in parallel programminga basic knowledge of SPARC assembly (desirable)Unless otherwise noted, topics in this book are not limited to a programming language, parallelization method, or software version. However, emphasis is on techniques relevant to applications written in Fortran 77, Fortran 90, and C, because these languages are most commonly used in HPC and technical applications.Most topics can be applied to C++ programs; however, we do not address performance optimization issues for object oriented programming. We refer readers to other resources.How This Book Is OrganizedThis book presents information so that it follows logical stages of the process for application development and optimization. We pay special attention to issues related to parallel applications and to using appropriate performance monitoring tools.Wherever applicable, sections are illustrated with code examples that show benefits of methods described.Part I - Getting StartedChapter 1 "Introduction," introduces optimization for HPC applications. We describe the basics of the optimization process and illustrate it with flow charts for serial and parallel optimization.Chapter 2 "Overview of Sun UltraSPARC Solaris Platforms," describes the available "tools of trade" for HPC developers using Solaris platforms. It gives an overview of Sun hardware and software products for technical computing. Also, the chapter introduces software development tools.Chapter 3 "Application Development on Solaris," considers development and porting issues on Sun platforms. It includes sections on binary compatibility between platforms, standards conformance, code verification tools, language interoperability, and 64-bit porting issues.Part II - Optimizing Serial ApplicationsChapter 4 "Measuring Program Performance," focuses on tools that measure application performance. Accurate measurement of performance is crucial in tuning. We describe accurate timers available on Solaris, profiling tools, Forte Developer 6 Performance Analyzer, hardware performance counter access tools on UltraSPARC processors, and other system monitoring tools.Chapter 5 "Basic Compiler Optimizations," introduces basic compiler optimizations and how to use compiler flags correctly. Options covered in this chapter are safe and generally can be applied without knowledge of any specifics of the application. The impact of using these flags is illustrated with examples, and analysis of the generated code with and without the options is presented.Chapter 6 "Advanced Compiler Optimizations," extends Chapter 5 and gives an overview of techniques that enable aggressive compiler optimizations. These often result in additional performance gains but may also lead to incorrect answers or spurious side-effects. Also, we cover performance related compiler pragmas and directives, which can be inserted in a program. Information about a program can be passed to the compiler, allowing additional optimizations.Chapter 7 "Linker and Libraries in Performance Optimization," highlights optimized libraries and features of the Solaris linker that can be used for application optimization. We describe the platform-specific optimized math libraries whose use can result in significant performance gains. We show linker techniques that allow linking of these platform-specific libraries in a portable fashion.Chapter 8 "Source Code Optimization," provides an overview of tuning techniques at the source code level. The techniques were selected from the point of view of better utilizing the underlying architectural features of UltraSPARC systems. We pay special attention to memory hierarchy utilization such as cache blocking and reducing the translation lookaside buffer (TLB) misses. We present ways of simplifying the code to allow better compiler optimizations, such as alias disambiguation in C programs, to take place.Chapter 9 "Loop Optimization," focuses on optimizing loops, one of the most commonly used constructs in scientific and HPC programs. We discuss ways in which developers can help the compiler control loop fusion and fission, as well as perform loop peeling. We show examples of register-tiling and consider loops with branches.Part III - Optimizing Parallel ApplicationsChapter 10 "Parallel Processing Models on Solaris," introduces concepts of parallel programming and different parallelization models available on Solaris/SPARC systems: automatic compiler parallelization, directives-based parallelism, explicit multithreading, UNIX fork/exec, message passing model, and hybrid programming (combined directives and message-passing).Chapter 11 "Parallel Performance Measurement Tools," details the tools for performance measurement and monitoring of parallel programs. Similar to Chapter 4, we focus on accurate timers for timing parallel programs, tools for measuring synchronization and communication overheads, tools for measuring hardware counters, and tools for multiprocessor system monitoring.Chapter 12 "Optimization of Explicitly Threaded Programs," provides an overview of explicit multithreading of programs using P-threads and Solaris threads. An overview of thread scheduling models in Solaris and their relevance to HPC programs is given and techniques for decreasing synchronization overheads are described.Chapter 13 "Optimization of Programs Using Compiler Parallelization," covers support and optimization techniques for automatic and directive-based parallelization in Sun compilers. Special emphasis is given to tuning OpenMP programs using the Fortran 95 compiler. OpenMP programming styles and data-scoping issues are illustrated with examples. Comparisons between OpenMP and P-threads approaches are presented.Chapter 14 "Optimization of Message-Passing Programs," describes message-passing models and how to tune MPI programs. We present an overview of message-passing programming models, compiling and linking programs using Sun MPI, and using Sun MPI environment variables. This chapter describes approaches for optimizing point-to-point and global communication with Sun MPI, using the S3L scientific library and using a hybrid OpenMP/MPI model.Part IV - AppendicesAppendix A "Commands That Identify System Configuration Parameters," lists useful Solaris commands that identify system configuration parameters.Appendix B "Architecture of UltraSPARC Microprocessor Family," gives an overview of architectural features of the UltraSPARC microprocessor family.Appendix C "Architecture of UltraSPARC Interconnect Family," describes the architecture of interconnect technologies for UltraSPARC systems.Appendix D "Hardware Counter Performance Metrics," shares some useful performance metrics that can be derived from hardware counters on UltraSPARC systems.Appendix E "Interval Arithmetic Support in Forte Developer 6 Fortran 95 Compiler," gives an overview of interval arithmetic support in the Forte Developer 6 Fortran 95 compiler.Appendix F "Differences in I/O Performance," considers the performance of different I/O techniques.Additional ResourcesTo keep the scope of this book manageable, we intentionally omitted many subjects related to performance optimization. Our criteria was to omit subjects that were not applicable to a wide range of applications. Many of these subjects are presented in other documentation for Sun products. The following is a list of publications you may find useful for more narrowly focused subjects:Numerical Computation GuideFortran Programming GuideFortran User's GuideAnalyzing Program Performance with Sun WorkshopC User's GuideForte 6 update 1 C User's Guide SupplementLinker and Libraries GuideSun Performance Library ReferenceMultithreaded Programming GuideProgramming Utilities Guide64-bit Developer's GuideSolaris Tunable Parameters Reference ManualSun HPC ClusterTools 3.1 Performance GuideSun MPI 4.1 Programming and Reference GuidePrism 6.1 User's GuideSun HPC ClusterTools 3.1 Installation GuideAll these publications are available online at http://docs.sun.com. We strongly recommend that developers visit this site, because nearly all published Sun documentation is available there. Printed versions are available from Sun Documentation Center at Fatbrain:http://www1.fatbrain.com/documentation/sunThe following publications are related to UltraSPARC microprocessors:UltraSPARC I and II User's ManualUltraSPARC IIi User's ManualThese publications are available at:http://www.sun.com/microelectronics/manualsOther sites of great use for developers are http://www.sun.com/developers/ and http://soldc.sun.com/, which contains current information for the Sun developer community.A description of Sun product lines is available at http://www.sun.com/desktop and http://www.sun.com/servers. Sun products and solutions for the HPC are listed at http://www.sun.com/hpc.For specialized books and additional theoretical information on application optimization, we refer readers to other sources. There are many excellent books on topics such as optimizing compilers, software tuning techniques, and efficient parallelization. The following are some helpful resources:J. Hennessy, D. Patterson - Computer Architecture: A Quantitative Approach, Second Edition; Morgan Kaufmann Publishing, 1996K. Dowd, Ch. Severance - High Performance Computing, Second Edition; O'Reilly & Associates, 1998D. E. Culler, J. P. Singh, A. Gupta - Parallel Computer Architecture: A Hardware Software Approach, Morgan Kaufmann Publishing, 1999S. S. Muchnick - Advanced Compiler Design and Implementation; Morgan Kaufmann Publishing, 1997.S. Kleinman, D. Shah, B. Smaalders - Programming with Threads; SunSoft Press, A Prentice Hall Title, 1996.W. Gropp, E. Lusk, A. Skjellum - Using MPI: Portable Parallel Programming with Message-Passing Interface (Scientific and Engineering Computation Series); Second Edition, MIT Press, 1999.R. Chandra, L. Dagum, D. Kohr, D. Maydan, J. McDonald, R. Menon - Parallel Programming in OpenMP; Morgan Kaufmann Publishing, 2000.More resources are listed in the References section at the end of this book.Readers interested in "hands on" training should check with the Sun Educational Services to determine if a class is scheduled and enrollment is open.Code ExamplesMany sections in this book are illustrated with code examples that show benefits of optimization techniques and coding practices. The code examples can be downloaded from the Sun BluePrints site:http://www.sun.com/blueprints/toolsA makefile is provided for each chapter so that the examples can be run with a single make command.Unless otherwise noted, all examples and results presented in this book use Forte Developer 6 compilers and the HPC 3.1 ClusterTools release. The results for serial runs were obtained on Sun Ultra 60, Sun Ultra 80, and Sun Blade 1000 systems. The results for parallel runs were performed on Sun Enterprise 4500, Sun Enterprise 10000 servers, and a Sun technical compute farm. The system parameters are listed in TABLE P-1 and TABLE P-2, respectively. More information about Sun platforms is in Chapter 2.