CoSA: Scheduling by Constrained Optimization for Spatial Accelerators

Abstract

Recent advances in Deep Neural Networks (DNNs) have led to active development of specialized DNN accelerators, many of which feature a large number of processing elements laid out spatially, together with a multi-level memory hierarchy and flexible interconnect. While DNN accelerators can take advantage of data reuse and achieve high peak throughput, they also expose a large number of runtime parameters to the programmers who need to explicitly manage how computation is scheduled both spatially and temporally. In fact, different scheduling choices can lead to wide variations in performance and efficiency, motivating the need for a fast and efficient search strategy to navigate the vast scheduling space.To address this challenge, we present CoSA, a constrainedoptimization-based approach for scheduling DNN accelerators. As opposed to existing approaches that either rely on designers' heuristics or iterative methods to navigate the search space, CoSA expresses scheduling decisions as a constrained-optimization problem that can be deterministically solved using mathematical optimization techniques. Specifically, CoSA leverages the regularities in DNN operators and hardware to formulate the DNN scheduling space into a mixed-integer programming (MIP) problem with algorithmic and architectural constraints, which can be solved to automatically generate a highly efficient schedule in one shot. We demonstrate that CoSA-generated schedules significantly outperform state-of-the-art approaches by a geometric mean of up to 2.5× across a wide range of DNN networks while improving the time-to-solution by 90×.

Search for Optimal Systolic Arrays: A Comprehensive Automated Exploration Framework and Lessons Learned

“…Mathematical programming-based methods. These methods formulate the search task as a mathematical optimization problem and resolve it with optimization solvers [4,17,26,27]. The recent work CoSA [17] modeled the design space as a mixed integer programming (MIP) problem.…”

“…There is a plethora of previous works on performance tuning of systolic arrays [4,9,15,17,19,21,28,31,41]. Table 1 lists several recent works.…”

“…In addition, in this work, we cover non-divisor tiling factors in loop tiling. Many previous works only used divisors for simplicity [4,9,17,31,41]. However, this choice could lead to significant performance loss, as discussed in Section 1.…”

“…Many previous works have attempted this challenging task by looking into different dimensions of the design space and proposing various auto-tuning methods [4,9,17,19,21,28,31,41]. However, after a thorough examination of the previous works, we identified several limitations that need to be addressed.…”

“…Limitation 1: Incomplete coverage of the design space. When selecting the tiling factors, many previous works only considered divisors to reduce the design space [4,9,17,41]. In Figure 1, we compare the throughput and DSP usage of best systolic arrays found when limiting tiling factors to 1) divisors only (Design 1) and 2) both divisors and non-divisors (Design 4) for a 1024 × 1024 × 1024 matrix multiplication (MM) 2 .…”