ITT-RNA: Imperfection Tolerable Training for RRAM-Crossbar-Based Deep Neural-Network Accelerator

Song, Zhuoran; Sun, Yanan; Chen, Lerong; Li, Tianjian; Jing, Naifeng; Liang, Xiaoyao; Jiāng, Lì

doi:10.1109/tcad.2020.2989373

Cited by 20 publications

(7 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…[25,26] Addtionally, process variation and stuck-at fault errors have been widely reported to cause performance degradation, although can be partially compensated by various methods with extra costs. [27][28][29][30][31][32][33][34][35][36][37][38][39] These issues can be generally attributed to the nonbiological conventional learning algorithm of DNN, that is, error backpropagation-based gradient descent weight update, [40,41] which needs both the VMMs and the conductance tuning in high precision. [42][43][44] Novel neural network structures and learning algorithms need to be explored to address these issues.…”

Section: Doi: 101002/aisy202100249mentioning

confidence: 99%

Efficient Training of the Memristive Deep Belief Net Immune to Non‐Idealities of the Synaptic Devices

Wang

Hoffer

Greenberg-Toledo

et al. 2022

Advanced Intelligent Systems

View full text Add to dashboard Cite

The tunability of conductance states of various emerging non-volatile memristive devices emulates the plasticity of biological synapses, making it promising in the hardware realization of large-scale neuromorphic systems. The inference of the neural network can be greatly accelerated by the vector-matrix multiplication (VMM) performed within a crossbar array of memristive devices in one step. Nevertheless, the implementation of the VMM needs complex peripheral circuits and the complexity further increases since non-idealities of memristive devices prevent precise conductance tuning (especially for the online training) and largely degrade the performance of the deep neural networks (DNNs). Here, we present an efficient online training method of the memristive deep belief net (DBN). The proposed memristive DBN uses stochastically binarized activations, reducing the complexity of peripheral circuits, and uses the contrastive divergence (CD) based gradient descent learning algorithm. The analog VMM and digital CD are performed separately in a mixed-signal hardware arrangement, making the memristive DBN high immune to non-idealities of synaptic devices. The number of write operations on memristive devices is reduced by two orders of magnitude. The recognition accuracy of 95%~97% can be achieved for the MNIST dataset using pulsed synaptic behaviors of various memristive synaptic devices.

show abstract

Section: Doi: 101002/aisy202100249mentioning

confidence: 99%

Efficient Training of the Memristive Deep Belief Net Immune to Non‐Idealities of the Synaptic Devices

Wang

Hoffer

Greenberg-Toledo

et al. 2022

Advanced Intelligent Systems

View full text Add to dashboard Cite

show abstract

“…Some efficient methods have been recently proposed to detect the types and locations of faults in memristor crossbars, but they mainly focused on hard faults, while soft faults were unaddressed. [20,34,35] Song et al [19] proposed to detect the conductance variation (i.e., soft fault) of every memristor and record it together with the location of this memristor in a buffer. This bit-wise detection method is relatively slow and the stored information of every memristor's conductance variation value is redundant.…”

Section: Multifault Detectionmentioning

confidence: 99%

“…Liu et al [ 16 ] applied a fault‐aware retraining scheme to SLP on the MNIST dataset (note: the MNIST dataset is always used hereafter unless otherwise specified), recovering the accuracy to 98.1% of the ideal value in the presence of 20% hard faults. Chen and Song et al [ 15,19 ] used a bipartile‐matching algorithm to map significant weights to the fault‐free memristors. They further reduced the large weights mapped onto the faulty memristors and then performed retraining.…”

Section: Preliminariesmentioning

confidence: 99%

“…Therefore, our method appears to be simpler and more efficient than the previous fault detection methods when dealing with multifaults. [ 19,20 ]…”

Section: Mftt Schemementioning

confidence: 99%

“…On the other hand, the software‐level solutions typically contain some or all of the following steps: adaptive weight mapping that maps the significant weights onto the fault‐free memristors, weight pruning that fixes the weights mapped onto the faulty memristors to zero (or reducing these weights), and ex situ training (or retraining). [ 18–20 ] The software‐level solutions can compensate the hard faults‐induced accuracy losses by taking advantage of the inherent sparsity of NN. In addition, they can save the hardware overhead.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

A Multifault‐Tolerant Training Scheme for Nonideal Memristive Neural Networks

Chen

Fan

Dong

et al. 2022

Advanced Intelligent Systems

View full text Add to dashboard Cite

Memristor crossbar is extensively investigated as an energy‐efficient accelerator for neural network (NN) computations. However, hardware implementation of NNs using realistic memristors is challenging due to the ubiquity of faults (mainly classified into hard and soft faults) in memristors. Herein, a hardware‐friendly, low‐power multifault‐tolerant training (MFTT) scheme capable of addressing both hard and soft faults simultaneously for memristive NNs is proposed. The MFTT scheme consists of multifault detection, targeted weight pruning, and in situ training with the Manhattan update rule. Specifically, multifault detection is first conducted to detect both hard and large soft faults. The detected faulty weights are subsequently pruned to prevent them from disturbing the NN. The sparsified NN after pruning is in situ trained so that the hard and large soft faults can be effectively tolerated using the sparsity and self‐adaptivity of NNs. In addition, the remaining small soft faults can be well tolerated by the Manhattan update rule. Experimentally, MFTT demonstrates the lowest accuracy losses among several representative fault‐tolerant schemes not only in the hard fault‐only (when the hard fault ratio exceeds 10%) and soft fault‐only (when the soft faults are large) cases, but also in the case where both types of faults coexist.

show abstract

Mitigating the Effects of RRAM Process Variation on the Accuracy of Artificial Neural Networks

Fritscher¹,

Knödtel²,

Mallah

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

ITT-RNA: Imperfection Tolerable Training for RRAM-Crossbar-Based Deep Neural-Network Accelerator

Cited by 20 publications

References 35 publications

Efficient Training of the Memristive Deep Belief Net Immune to Non‐Idealities of the Synaptic Devices

Efficient Training of the Memristive Deep Belief Net Immune to Non‐Idealities of the Synaptic Devices

A Multifault‐Tolerant Training Scheme for Nonideal Memristive Neural Networks

Mitigating the Effects of RRAM Process Variation on the Accuracy of Artificial Neural Networks

Contact Info

Product

Resources

About