Communication Characterization and Optimization of Applications Using Topology-Aware Task Mapping on Large Supercomputers

Abstract

On large supercomputers, the job scheduling systems may assign a non-contiguous node allocation for user applications depending on available resources. With parallel applications using MPI (Message Passing Interface), the default process ordering does not take into account the actual physical node layout available to the application. This contributes to non-locality in terms of physical network topology and impacts communication performance of the application. In order to mitigate such performance penalties, this work describes techniques to identify suitable task mapping that takes the layout of the allocated nodes as well as the application's communication behavior into account. During the first phase of this research, we instrumented and collected performance data to characterize communication behavior of critical US DOE (United States-Department of Energy) applications using an augmented version of the mpiP tool. Subsequently, we developed several reordering methods (spectral bisection, neighbor join tree etc.) to combine node layout and application communication data for optimized task placement. We developed a tool called mpiAproxy to facilitate detailed evaluation of the various reordering algorithms without requiring full application executions. This work presents a comprehensive performance evaluation (14,000 experiments) of the various task mapping techniques in lowering communication costs on Titan, the leadership class supercomputer at Oak Ridge National Laboratory. Publication rights licensed to ACM. ACM acknowledges that this contribution was authored or co-authored by an employee, contractor or affiliate of the United States government. As such, the Government retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, for Government purposes only.

Reproducibility in Benchmarking Parallel Fast Fourier Transform based Applications

“…Figure 1 demonstrates runtime variability when running fast Fourier Transform programs [9,10] for solving the Navier-Stokes equations on Hazelhen and Shaheen II. Both these computers use adaptive routing and job placement to speed up job throughput [18], but this can result in some run time variability, a subject of current research [7,28,35,38,39].…”

Node aware sparse matrix–vector multiplication

“…As a result, hypergraph partitioning also leads to a reduction in parallel communication requirements, albeit at a larger one-time setup cost. Topology-aware task mapping is used to accurately map partitions to the allocated nodes of a supercomputer, reducing the overall cost associated with communication [11,12,13,14,15]. The approach introduced in this paper complements these efforts by providing an additional level of optimization in handling communication.Topology-aware methods and aggregation of data are commmonly used to reduce communication costs, particularly in collective operations [16,17,18,19].…”