Performance modeling for SPMD message-passing programs

Abstract

Today's massively parallel machines are typically message-passing systems consisting of hundreds or thousands of processors. Implementing parallel applications efficiently in this environment is a challenging task, and poor parallel design decisions can be expensive to correct. Tools and techniques that allow the fast and accurate evaluation of different parallelization strategies would significantly improve the productivity of application developers and increase throughput on pardel architectures. This paper investigates one of the major issues in building tools to compare parallelization strategies: determining what type of performance models of the application code and of the computer system are sufficient for a fast and accurate comparison of different strategies. The paper is built around a case study employing the Performance Prediction Tool (PerPreT) to predict performance of the Parallel Spectral Transform Shallow Water Model code (PSTSWM) on the Intel Paragon.PSTSWM is a parallel application code that was designed to evaluate different pard e l strategies for the spectral transform method as it is used in climate modeling and weather forecasting. Multiple parallel algorithms and algorithm variants are embedded in the code. PerPreT uses a relatively simple algebraic model to predict execution time for SPMD (Single Program Multiple Data) parallel applications. Applications are modeled through parameterized formulae for communication and computation, where the parameters include the problem size, the number of processors used to execute the program, and system characteristics (e+, setup times for communication, link bandwidth, and sustained computing performance per processor).In this paper we describe performance models that predict the performance of the different algorithms in PSTSWM accurately enough to allow them to be compared, establishing the feasibility of such a demanding application of performance modeling. We also discuss issues in generating and validating the performance models, emphasizing the practical importance of tools such as PerPreT in such studies.

Methodology for modelling SPMD hybrid parallel computation

“…Memory access behaviour can strongly affect floating-point performance as well as interaction with the communication network. Quantification of thread parameters is one point of distinction between this work and other analytic modelling efforts [18,21].…”

“…Brehm et al [21] developed a tool for analytic model evaluation with a focus on assisted parallelization. Therefore, they emphasize determination of the appropriate level of detail for the application and for the computer system.…”

Inferring Large-Scale Computation Behavior via Trace Extrapolation

“…Code modification must be made in the form of adding annotations or "modeling assertions" around key application constructs, which limits the scalability of this approach for large scale HPC applications that have many thousands of lines of code. Various other researchers [11], [12], [13], [14] have also used application-specific approaches to generate performance models, however in general they are difficult to automate and generalize because they require extensive guidance from domain experts.…”

Design and performance evaluation of a portable parallel library for space-time adaptive processing

“…Models of individual modules are combined to produce models of the overall execution time. Brehm et al [32] showed that a similar approach produced good models of a message-passing parallel program used to solve the nonlinear shallow water equations.…”