A Scalable Multi-TeraOPS Core for AI Training and Inference

Shukla, Sunil; Fleischer, Bruce; Ziegler, Matthew M.; Silberman, J. A.; Oh, Jinwook; Srinivasan, Vijayalakshmi; Choi, Jungwook; Mueller, Silvia Melitta; Agrawal, Ankur; Babinsky, Tina; Cao, Nianwen; Chen, Chia-Yu; Chuang, Pierce; Fox, Thomas W.; Gristede, George; Guillorn, Michael A.; Haynie, Howard; Klaiber, Michael J.; Lee, Dongsoo; Lo, Shih-Hsien; Maier, Gary; Scheuermann, M.; Venkataramani, Swagath; Vezyrtzis, Christos; Wang, Naigang; Yee, Fanchieh; Zhou, Ching; Lu, Pong‐Fei; Curran, Brian; Chang, Leland; Gopalakrishnan, Kailash

doi:10.1109/lssc.2019.2902738

Cited by 29 publications

(19 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Analysis of machine-learning workloads for both training and inference has shown that high-dimensionality MVMs dominate, accounting for 70%-95% of the computation [22]. This is especially true for the CNNs of interest in target edge applications.…”

Section: Architecture Overview and Rationalementioning

confidence: 99%

“…These include element-wise operations, such as activation functions, scaling, adding, and offset. [22], as well as other signal-processing operations required in audio, video, and so on, and pipelines [2], [23], [24]. Such computations are distinct from MVMs in that they benefit from significant data locality (i.e., a small number of operands is involved in the fundamental operations even though the operations might be parallelized).…”

Section: Architecture Overview and Rationalementioning

confidence: 99%

See 1 more Smart Citation

A Programmable Heterogeneous Microprocessor Based on Bit-Scalable In-Memory Computing

Jia

Valavi

Tang

et al. 2020

IEEE J. Solid-State Circuits

144

View full text Add to dashboard Cite

In-memory computing (IMC) addresses the cost of accessing data from memory in a manner that introduces a tradeoff between energy/throughput and computation signal-tonoise ratio (SNR). However, low SNR posed a primary restriction to integrating IMC in larger, heterogeneous architectures required for practical workloads due to the challenges with creating robust abstractions necessary for the hardware and software stack. This work exploits recent progress in high-SNR IMC to achieve a programmable heterogeneous microprocessor architecture implemented in 65-nm CMOS and corresponding interfaces to the software that enables mapping of application workloads. The architecture consists of a 590-Kb IMC accelerator, configurable digital near-memory-computing (NMC) accelerator, RISC-V CPU, and other peripherals. To enable programmability, microarchitectural design of the IMC accelerator provides the integration in the standard processor memory space, areaand energy-efficient analog-to-digital conversion for interfacing to NMC, bit-scalable computation (1-8 b), and input-vector sparsity-proportional energy consumption. The IMC accelerator demonstrates excellent matching between computed outputs and idealized software-modeled outputs, at 1b TOPS/W of 192|400 and 1b-TOPS/mm 2 of 0.60|0.24 for MAC hardware, at V D D of 1.2|0.85 V, both of which scale directly with the bit precision of the input vector and matrix elements. Software libraries developed for application mapping are used to demonstrate CIFAR-10 image classification with a ten-layer CNN, achieving accuracy, throughput, and energy of 89.3%|92.4%, 176|23 images/s, and 5.31|105.2 µJ/image, for 1|4 b quantization levels. Index Terms-Charge-domain compute, deep learning, hardware accelerators, in-memory computing (IMC), neural networks (NNs). I. INTRODUCTION M ACHINE-LEARNING inference, particularly based on neural networks (NNs), has provided unprecedented capabilities in various cognitive tasks, such as vision and language processing. [1]-[5]. However, pervasive deployment, especially in edge applications, has been limited by the high computational requirements that are dominated by highdimensionality matrix-vector multiplications (MVMs). To address this, many optimizations, focusing on both hardware specialization and model design (e.g., sparsity, compression,

show abstract

Section: Architecture Overview and Rationalementioning

confidence: 99%

Section: Architecture Overview and Rationalementioning

confidence: 99%

A Programmable Heterogeneous Microprocessor Based on Bit-Scalable In-Memory Computing

Jia

Valavi

Tang

et al. 2020

IEEE J. Solid-State Circuits

144

View full text Add to dashboard Cite

show abstract

“…[114] The 1T1R array is 64 kb (256 Â 256) with RRAM devices in binary states. [115] The spike-based ISNA takes a different approach to enhance the energy efficiency by lowering the power consumption at the cost of %200 ns latency. In the memory mode, the read/write logics, drivers, and amplifiers realize data programming and sensing.…”

Section: Spike-based Designmentioning

confidence: 99%

Resistive Memory‐Based In‐Memory Computing: From Device and Large‐Scale Integration System Perspectives

Yan

Qiao

et al. 2019

Advanced Intelligent Systems

View full text Add to dashboard Cite

In‐memory computing is a computing scheme that integrates data storage and arithmetic computation functions. Resistive random access memory (RRAM) arrays with innovative peripheral circuitry provide the capability of performing vector‐matrix multiplication beyond the basic Boolean logic. With such a memory–computation duality, RRAM‐based in‐memory computing enables an efficient hardware solution for matrix‐multiplication‐dependent neural networks and related applications. Herein, the recent development of RRAM nanoscale devices and the parallel progress on circuit and microarchitecture layers are discussed. Well suited for analog synapse and neuron implementation, RRAM device properties and characteristics are emphasized herein. 3D‐stackable RRAM and on‐chip training are introduced in large‐scale integration. The circuit design and system organization of RRAM‐based in‐memory computing are essential to breaking the von Neumann bottleneck. These outcomes illuminate the way for the large‐scale implementation of ultra‐low‐power and dense neural network accelerators.

show abstract

“…Even BNNs that use binarized weights during inference require floating-point computations for their training [6]. Several studies on training hardware using the backpropagation algorithm have recently been reported [10]- [12]. Training using a hardware accelerator [10], [11] shows insufficient improvement in terms of latency compared to the improvement obtained for inference carried out on the same accelerator.…”

Section: Introductionmentioning

confidence: 99%

“…Several studies on training hardware using the backpropagation algorithm have recently been reported [10]- [12]. Training using a hardware accelerator [10], [11] shows insufficient improvement in terms of latency compared to the improvement obtained for inference carried out on the same accelerator. Moreover, these works require floatingpoint arithmetic during training even though they represent low-precision data during inference.…”

Section: Introductionmentioning

confidence: 99%

Training Hardware for Binarized Convolutional Neural Network Based on CMOS Invertible Logic

et al. 2020

View full text Add to dashboard Cite

In this paper, we implement fast and power-efficient training hardware for convolutional neural networks (CNNs) based on CMOS invertible logic. The backpropagation algorithm is generally hard to implement in hardware because it requires high-precision floating-point arithmetic. Even though parameters of CNNs can be represented by fixed points or even binary during inference, it is still represented by floating points during training. Our hardware uses low-precision data representation for both inference and training. For hardware implementation, we exploit CMOS invertible logic for training. The use of invertible logic enables logic circuits to compute probabilistic bidirectional operation (forward and backward modes) and can be implemented by stochastic computing. The proposed hardware obtains parameters of neural networks such as weights directly from given data (an input feature map and a true label) without backpropagation. For performance evaluation, the proposed hardware is implemented on an FPGA and trains a binarized 2-layer convolutional neural network model using a modified MNIST dataset. This implementation shows an energy efficiency improvement of approximately 134x compared to that of a CPU implementation that executes the training of the same model as that used in the proposed hardware. Training on the proposed hardware is approximately 40x faster than training on the CPU using the backpropagation algorithm while maintaining almost the same cognition accuracy. INDEX TERMS CMOS invertible logic, Neural network, Stochastic computing.

show abstract

A Scalable Multi-TeraOPS Core for AI Training and Inference

Cited by 29 publications

References 6 publications

A Programmable Heterogeneous Microprocessor Based on Bit-Scalable In-Memory Computing

A Programmable Heterogeneous Microprocessor Based on Bit-Scalable In-Memory Computing

Resistive Memory‐Based In‐Memory Computing: From Device and Large‐Scale Integration System Perspectives

Training Hardware for Binarized Convolutional Neural Network Based on CMOS Invertible Logic

Contact Info

Product

Resources

About