SSM: a high-performance scheme for in situ training of imprecise memristor neural networks

Wang, Yaoyuan; Wu, Shuang; Tian, Lei; Shi, Luping

doi:10.1016/j.neucom.2020.04.130

Cited by 17 publications

(12 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…in situ (re)training of weights (or just a subset of them [ 14 ] ) to recover from the effects of nonidealities, [ 15 , 16 , 17 , 18 , 19 ] including in convolutional neural networks (CNNs), [ 14 , 20 ] recurrent structures, [ 20 ] and in neural networks used for reinforcement learning [ 21 ]…”

Section: Introductionmentioning

confidence: 99%

Nonideality‐Aware Training for Accurate and Robust Low‐Power Memristive Neural Networks

et al. 2022

View full text Add to dashboard Cite

Recent years have seen a rapid rise of artificial neural networks being employed in a number of cognitive tasks. The ever‐increasing computing requirements of these structures have contributed to a desire for novel technologies and paradigms, including memristor‐based hardware accelerators. Solutions based on memristive crossbars and analog data processing promise to improve the overall energy efficiency. However, memristor nonidealities can lead to the degradation of neural network accuracy, while the attempts to mitigate these negative effects often introduce design trade‐offs, such as those between power and reliability. In this work, authors design nonideality‐aware training of memristor‐based neural networks capable of dealing with the most common device nonidealities. The feasibility of using high‐resistance devices that exhibit high I‐V nonlinearity is demonstrated—by analyzing experimental data and employing nonideality‐aware training, it is estimated that the energy efficiency of memristive vector‐matrix multipliers is improved by almost three orders of magnitude (0.715 TOPs−1W−1 to 381 TOPs−1W−1) while maintaining similar accuracy. It is shown that associating the parameters of neural networks with individual memristors allows to bias these devices toward less conductive states through regularization of the corresponding optimization problem, while modifying the validation procedure leads to more reliable estimates of performance. The authors demonstrate the universality and robustness of this approach when dealing with a wide range of nonidealities.

show abstract

Section: Introductionmentioning

confidence: 99%

Nonideality‐Aware Training for Accurate and Robust Low‐Power Memristive Neural Networks

et al. 2022

View full text Add to dashboard Cite

show abstract

“…But, even with technological maturity, other imperfections may remain due to their inseparable relationship with the physics of the device/circuit (e.g., programming variability and interconnect resistance). Numerous methods have therefore been proposed to mitigate the effect of these non-idealities for NN applications Lim et al, 2018;He et al, 2019;Liu et al, 2020a;Wang et al, 2020b;Mahmoodi et al, 2020;Pan et al, 2020;Zhang et al, 2020;Xi et al, 2021). Although a mitigation approach can significantly increase the performance of RSM-based NNs, none of these methods are able to achieve the accuracy of their software counterparts.…”

Section: Introductionmentioning

confidence: 99%

Exploiting Non-idealities of Resistive Switching Memories for Efficient Machine Learning

et al. 2022

View full text Add to dashboard Cite

Novel computing architectures based on resistive switching memories (also known as memristors or RRAMs) have been shown to be promising approaches for tackling the energy inefficiency of deep learning and spiking neural networks. However, resistive switch technology is immature and suffers from numerous imperfections, which are often considered limitations on implementations of artificial neural networks. Nevertheless, a reasonable amount of variability can be harnessed to implement efficient probabilistic or approximate computing. This approach turns out to improve robustness, decrease overfitting and reduce energy consumption for specific applications, such as Bayesian and spiking neural networks. Thus, certain non-idealities could become opportunities if we adapt machine learning methods to the intrinsic characteristics of resistive switching memories. In this short review, we introduce some key considerations for circuit design and the most common non-idealities. We illustrate the possible benefits of stochasticity and compression with examples of well-established software methods. We then present an overview of recent neural network implementations that exploit the imperfections of resistive switching memory, and discuss the potential and limitations of these approaches.

show abstract

“…For the tolerance of soft faults, the in situ training which can adjust the weights self‐adaptively on chip is an effective approach. [ 21 ] The fault tolerance can be further improved using some modified weight update rules, such as the Manhattan update rule, [ 22,23 ] stochastic update rule, [ 6,24 ] stochastic sparse update with momentum adaption, [ 25 ] and sign‐based backpropagation (BP) algorithm. [ 26,27 ] The main idea of these modified weight update rules is to move the weights in the general direction that results in the reduction of loss function without regulating the step size.…”

Section: Introductionmentioning

confidence: 99%

A Multifault‐Tolerant Training Scheme for Nonideal Memristive Neural Networks

Chen

Fan

Dong

et al. 2022

Advanced Intelligent Systems

View full text Add to dashboard Cite

Memristor crossbar is extensively investigated as an energy‐efficient accelerator for neural network (NN) computations. However, hardware implementation of NNs using realistic memristors is challenging due to the ubiquity of faults (mainly classified into hard and soft faults) in memristors. Herein, a hardware‐friendly, low‐power multifault‐tolerant training (MFTT) scheme capable of addressing both hard and soft faults simultaneously for memristive NNs is proposed. The MFTT scheme consists of multifault detection, targeted weight pruning, and in situ training with the Manhattan update rule. Specifically, multifault detection is first conducted to detect both hard and large soft faults. The detected faulty weights are subsequently pruned to prevent them from disturbing the NN. The sparsified NN after pruning is in situ trained so that the hard and large soft faults can be effectively tolerated using the sparsity and self‐adaptivity of NNs. In addition, the remaining small soft faults can be well tolerated by the Manhattan update rule. Experimentally, MFTT demonstrates the lowest accuracy losses among several representative fault‐tolerant schemes not only in the hard fault‐only (when the hard fault ratio exceeds 10%) and soft fault‐only (when the soft faults are large) cases, but also in the case where both types of faults coexist.

show abstract

SSM: a high-performance scheme for in situ training of imprecise memristor neural networks

Cited by 17 publications

References 23 publications

Nonideality‐Aware Training for Accurate and Robust Low‐Power Memristive Neural Networks

Nonideality‐Aware Training for Accurate and Robust Low‐Power Memristive Neural Networks

Exploiting Non-idealities of Resistive Switching Memories for Efficient Machine Learning

A Multifault‐Tolerant Training Scheme for Nonideal Memristive Neural Networks

Contact Info

Product

Resources

About