Late Breaking Results: Building an On-Chip Deep Learning Memory Hierarchy Brick by Brick

Vivancos, Isak Edo; Sharify, Sayeh; Nikolić, Miloš; Bannon, Ciaran; Mahmoud, Magdi S.; Lascorz, Alberto Delmás; Moshovos, Andreas

doi:10.1109/dac18072.2020.9218728

Cited by 2 publications

(3 citation statements)

References 1 publication

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The synaptic weights in an SNN are updated based on local errors instead of global gradients backpropagated through layers, which is considered the key to neuromorphic hardware ( Boybat et al, 2018 ; Stewart et al, 2020 ). Neuromorphic hardware systems have become the core of hardware acceleration and embedded systems ( Vivancos et al, 2021 ).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Highly efficient neuromorphic learning system of spiking neural network with multi-compartment leaky integrate-and-fire neurons

Gao

Deng²,

Wang³

et al. 2022

Front. Neurosci.

View full text Add to dashboard Cite

A spiking neural network (SNN) is considered a high-performance learning system that matches the digital circuits and presents higher efficiency due to the architecture and computation of spiking neurons. While implementing a SNN on a field-programmable gate array (FPGA), the gradient back-propagation through layers consumes a surprising number of resources. In this paper, we aim to realize an efficient architecture of SNN on the FPGA to reduce resource and power consumption. The multi-compartment leaky integrate-and-fire (MLIF) model is used to convert spike trains to the plateau potential in dendrites. We accumulate the potential in the apical dendrite during the training period. The average of this accumulative result is the dendritic plateau potential and is used to guide the updates of synaptic weights. Based on this architecture, the SNN is implemented on FPGA efficiently. In the implementation of a neuromorphic learning system, the shift multiplier (shift MUL) module and piecewise linear (PWL) algorithm are used to replace multipliers and complex nonlinear functions to match the digital circuits. The neuromorphic learning system is constructed with resources on FPGA without dataflow between on-chip and off-chip memories. Our neuromorphic learning system performs with higher resource utilization and power efficiency than previous on-chip learning systems.

show abstract

Section: Introductionmentioning

confidence: 99%

“…One is the high-throughout data transmission between off-chip and on-chip memory. Although high-performance memory systems have been proposed to optimize data transmission, the power consumption is irreducible ( Lian et al, 2019 ; Vivancos et al, 2021 ). Dynamic random-access memory (DRAM) is usually used as the off-chip memory.…”

Section: Introductionmentioning

confidence: 99%

Highly efficient neuromorphic learning system of spiking neural network with multi-compartment leaky integrate-and-fire neurons

Gao

Deng²,

Wang³

et al. 2022

Front. Neurosci.

View full text Add to dashboard Cite

show abstract

“…Fundamentally, compression relies on non-uniformity in the data value distribution statically and temporally. It is instructive to compare and contrast the behavior of floating-point values during training with the fixed-point values commonly used during inference as many techniques have capitalized on the properties of the value stream during inference [36], [39]- [44]. During inference: 1) Zeros are very common especially in models using ReLU.…”

Section: Introductionmentioning

confidence: 99%

Schrödinger's FP: Dynamic Adaptation of Floating-Point Containers for Deep Learning Training

Nikolić¹,

Torres²,

Wang³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

We introduce a software-hardware co-design approach to reduce memory traffic and footprint during training with BFloat16 or FP32 boosting energy efficiency and execution time performance. We introduce methods to dynamically adjust the size and format of the floating-point containers used to store activations and weights during training. The different value distributions lead us to different approaches for exponents and mantissas. Gecko exploits the favourable exponent distribution with a loss-less delta encoding approach to reduce the total exponent footprint by up to 58% in comparison to a 32 bit floating point baseline. To content with the noisy mantissa distributions, we present two lossy methods to eliminate as many as possible least significant bits while not affecting accuracy. Quantum Mantissa, is a machine learning-first mantissa compression method that taps on training's gradient descent algorithm to also learn minimal mantissa bitlengths on a per-layer granularity, and obtain up to 92% reduction in total mantissa footprint. Alternatively, BitChop observes changes in the loss function during training to adjust mantissa bit-length network-wide yielding a reduction of 81% in footprint. Schrödinger's FP implements hardware encoders/decoders that guided by Gecko/Quantum Mantissa or Gecko/BitChop transparently encode/decode values when transferring to/from off-chip memory boosting energy efficiency and reducing execution time.

show abstract

Late Breaking Results: Building an On-Chip Deep Learning Memory Hierarchy Brick by Brick

Cited by 2 publications

References 1 publication

Highly efficient neuromorphic learning system of spiking neural network with multi-compartment leaky integrate-and-fire neurons

Highly efficient neuromorphic learning system of spiking neural network with multi-compartment leaky integrate-and-fire neurons

Schrödinger's FP: Dynamic Adaptation of Floating-Point Containers for Deep Learning Training

Contact Info

Product

Resources

About