Breaking the bandwidth wall in chip multiprocessors

Abstract

In throughput-aware CMPs like GPUs and DSPs, software-managed streaming memory systems are an effective way to tolerate high latencies. E.g., the Cell/B.E. incorporates local memories, and data transfers to/from those memories are overlapped with computation using DMAs. In such designs, the latency of the memory system has little impact on performance; instead, memory bandwidth becomes critical. With the increase in the number of cores, conventional DRAMs no longer suffice to satisfy the bandwidth demand. Hence, recent throughput-aware CMPs adopted caches to filter off-chip traffic. However, such caches are optimized for latency, not bandwidth.This work presents a re-design of the memory system in throughput-aware CMPs. Instead of a traditional latencyaware cache, we propose to spread the address space using fine-grained interleaving all over a shared non-coherent last-level cache (LLC). In this way, on-chip storage is optimally used, with no need to keep coherency. On the memory side, we also propose the use of interleaving across DRAMs but with a much finer granularity than usual pagesize approaches.Our proposal is highly optimized for bandwidth, not latency, by avoiding data replication in the LLC and by using fine-grained address space interleaving in both the LLC and the memory. For a CMP with 128 cores and 64-MB LLC, performance is improved by 21% due to the LLC optimizations and an extra 42% due to the off-chip memory optimizations, for a total 1.7 times performance improvement.

Collective Memory Transfers for Multi-Core Chips

“…Stencil calculations perform global sweeps through data structures that are typically much larger than the available data caches. As a result, data from main memory often cannot be transferred fast enough to avoid stalling the computational units on modern microprocessors [74,18,12,70,66]. Reorganizing these computations to fit into the caches has principally focused on tiling optimizations that exploit locality by performing operations on cache-sized blocks of data in each processor before moving on to the next block [56].…”

“…CMS has broad applicability because a wide variety of stencil-based kernels are memory bound [65,70,32,36]. Stencil-based kernels are also critical because they comprise the building blocks of applications ranging from image processing in consumer devices to the largest scale HPC applications such as climate modeling and fluid simulations.…”

“…In addition, numerous image processing applications are also memory bandwidth bound, such as the matrix-vector product in NVIDIA Fermi [1] and face recognition [38]. High graphics processing demands force the Xbox 360 to have 22.4GB/s of GDDR3 bandwidth to satisfy just three processors [70] Finally, CMS reduces memory access latency, which directly benefits latency-sensitive systems [66].…”

“…Memory bandwidth is not scaling rapidly enough to satisfy the increasing number of processors, making the performance of a wide variety of applications constrained by memory bandwidth [70,66,59,12,32,18,19,28,30,2,55]. In fact, current projections state that chip pins increase by 10% every year whereas on-chip processors double every 18 months [59].…”

“…Even SPEC benchmarks can saturate memory bandwidth in just eight-core CMPs [39]. In memory bandwidth-bound applications, techniques that increase memory bandwidth have a direct effect on execution time [65,70].…”