Coordinated CTA Combination and Bandwidth Partitioning for GPU Concurrent Kernel Execution

Abstract

Contemporary GPUs support multiple kernels to run concurrently on the same streaming multiprocessors (SMs). Recent studies have demonstrated that such concurrent kernel execution (CKE) improves both resource utilization and computational throughput. Most of the prior works focus on partitioning the GPU resources at the cooperative thread array (CTA) level or the warp scheduler level to improve CKE. However, significant performance slowdown and unfairness are observed when latency-sensitive kernels co-run with bandwidth-intensive ones. The reason is that bandwidth over-subscription from bandwidth-intensive kernels leads to much aggravated memory access latency, which is highly detrimental to latency-sensitive kernels. Even among bandwidth-intensive kernels, more intensive kernels may unfairly consume much higher bandwidth than less-intensive ones. In this article, we first make a case that such problems cannot be sufficiently solved by managing CTA combinations alone and reveal the fundamental reasons. Then, we propose a coordinated approach for CTA combination and bandwidth partitioning. Our approach dynamically detects co-running kernels as latency sensitive or bandwidth intensive. As both the DRAM bandwidth and L2-to-L1 Network-on-Chip (NoC) bandwidth can be the critical resource, our approach partitions both bandwidth resources coordinately along with selecting proper CTA combinations. The key objective is to allocate more CTA resources for latencysensitive kernels and more NoC/DRAM bandwidth resources to NoC-/DRAM-intensive kernels. We achieve it using a variation of dominant resource fairness (DRF). Compared with two state-of-the-art CKE optimization schemes, SMK [52] and WS [55], our approach improves the average harmonic speedup by 78% and 39%, respectively. Even compared to the best possible CTA combinations, which are obtained from an exhaustive search among all possible CTA combinations, our approach improves the harmonic speedup by up to 51% and 11% on average.

A Model-Based Software Solution for Simultaneous Multiple Kernels on GPUs

“…After the concurrent execution, the number of wavefronts remaining for k 1 is denoted as N wf _r emained (k 1 ), as shown in Equation (14).…”

“…Work [28] proposes an efficient bandwidth management for concurrent kernels, which is based on the patterns of an application's effective bandwidth changes with different thread-level parallelism. Lin et al [14] address the interferences of DRAM bandwidth and L2-to-L1 Networkon-Chip (NoC) bandwidth for corunning kernels and propose a coordinated approach to find an optimized SMK scheme and bandwidth partition for simultaneously running kernels.…”

“…The competition for dynamic resources among concurrent kernels within a CU has a great influence on performance. Work [14] shows that the bandwidth contentions of corunning kernels, including DRAM bandwidth and L2-to-L1 Networkon-Chip (NoC) bandwidth, play an important role in GPU multitasking. However, Maestro [22] and other work [14] both utilize architecture techniques to support dynamic resource management, which are impossible or generate great overheads by software solutions.…”

“…However, their LEFTOVER scheduling policy of GPU hardware schedulers [1,21] decreases the concurrency, because the first launched kernel may use up one of the static resources of a GPU and make other kernels unable to be dispatched. Solutions proposed in recent research to improve the concurrency of GPU kernels can be classified into two categories: (1) spatially partitioned sharing (SPS), which coexecutes different kernels on disjointed sets of compute units (CUs) [1,10,13,17], and (2) simultaneous multikernel (SMK), which runs multiple kernels simultaneously within a CU [14,21,22,29,31,33]. In general, SMK can improve the resource utilization even more than SPS, because SMK can launch more threads on a CU by corunning kernels with both complementary static resource requirements and interleaving instructions from kernels with low dynamic resource contentions, while SPS only allows different kernels to corun on disjointed sets of CUs [22].However, there is a lack of software solutions for applying SMK to GPUs.…”

“…The speedup, speedup, shows the performance improvement (Equation (20)). Harmonic speedup HS [10,14] is a balanced metric for both system throughput and fairness (Equation (21)), and weighted speedup W S [6,14] is a metric for system throughput (Equation (22)). For these three metrics, the higher one is the better one.…”