SparCE: Sparsity Aware General-Purpose Core Extensions to Accelerate Deep Neural Networks

Abstract

Deep Neural Networks (DNNs) have emerged as the method of choice for solving a wide range of machine learning tasks. Satiating the enormous growth in computational demand posed by DNNs is a key challenge for computing system designers and has most commonly been addressed through the design of custom accelerators. However, these specialized accelerators that utilize large quantities of multiply-accumulate units and on-chip memory are prohibitive in many design scenarios (e.g., wearable devices and IoT sensors), due to the stringent area/cost constraints. Therefore, accelerating DNNs on these low-power systems, comprising of mainly the indispensable general-purpose processor (GPP) cores, requires new approaches. In this work, we focus on improving the performance of DNNs on GPPs by exploiting a key attribute of DNNs, i.e. sparsity. We propose Sparsity aware Core Extensions (SPARCE) -a set of micro-architectural and ISA extensions that leverage sparsity and are minimally intrusive and low-overhead. We address the key challenges associated with exploiting sparsity in GPPs, viz., dynamically detecting whether an operand (e.g., the result of a load instruction) is zero and subsequently skipping a set of future instructions that use it. To maximize performance benefits, our design ensures that the instructions to be skipped are prevented from even being fetched, as squashing instructions comes with a penalty (e.g., a pipeline stall). SPARCE consists of 2 key micro-architectural enhancements. First, a Sparsity Register File (SpRF) is utilized to track registers that are zero. Next, a Sparsity aware Skip Address (SASA) table is used to indicate instruction sequences that can be skipped, and to associate specific SpRF registers to trigger instruction skipping. When an instruction is fetched, SPARCE dynamically pre-identifies whether the following instruction(s) can be skipped, and if so appropriately modifies the program counter, thereby skipping the redundant instructions and improving performance. We model SPARCE using the gem5 architectural simulator, and evaluate our approach on 6 state-of-the-art image-recognition DNNs in the context of both training and inference using the Caffe deep learning framework. On a scalar microprocessor, SPARCE achieves 1.11×-1.96× speedups across both convolution and fully-connected layers with 10%-90% sparsity, and 19%-31% reduction in execution time at the overall application-level. We also evaluate SPARCE on a 4-way SIMD ARMv8 processor using the OpenBLAS library, and demonstrate that SPARCE achieves 8%-15% reduction in the application-level execution time.

Democratizing IC Design: The Story of a New Movement and the Launch of the SSCS PICO Program [Society News]

“…The AFTx06 implements an RV32IM instruction set, integer multiplication and division, an integrated core local interrupt (CLINT) controller and a platform-level interrupt controller (PLIC), and it is expected to operate with a 100-MHz clock. A key feature of this microprocessor is the sparsity-aware core extensions (SparCE) architecture [2], a component that exploits sparsity in convolution arithmetic, allowing extraneous instructions (such as multiplication by zero; common in rectified linear unit functions) to be skipped at run time. The architecture has been designed to improve both the speed and power consumption of this common machine learning calculation.…”

SparseTrain

“…SparCE [40] skips ineffectual code blocks based on a sparse input. It annotates skippable code blocks in software and tests conditions in hardware.…”

“…Indeed, prior efforts spanning hardware to software and algorithms have exploited sparsity to eliminate computation or data transfers at different points in DNN computations. Most of these efforts, though, require hardware changes [3,7,13,34,38,40,57] and/or apply only to inference [3,7,13,15,34,35,50,53,57]. This is not ideal, since most of real-world DNN computations are performed on conventional CPUs and GPUs [4,16,33,51], and significant time goes into training.…”

“…Second, operating on sparse data incurs overhead: modern machines are highly optimized for dense computations, and suffer from the extra indirections and branches that appear when processing sparse data. Prior work either relies on custom hardware to minimize these overheads [3,7,13,34,40,57], or sophisticated pre-processing to "shape" the sparsity pattern to better match existing hardware [35,50,53]-which only applies to static sparsity.…”

Zeroploit

“…Finally, we compare our work on exploiting zeros in modern gaming applications with prior art on leveraging sparsity (i.e., memory loads and computations returning zeros) in deep neural networks (DNNs). Several researchers have exploited sparsity to improve DNN performance in accelerator architectures [1,34,50,65], GPUs [51], and general-purpose processors [56] through hardware enhancements. Like these efforts, Zeroploit seeks to improve performance by leveraging zero valued operands.…”