A quantization-aware regularized learning method in multilevel memristor-based neuromorphic computing system

Song, Chang; Liu, Beiye; Wen, Wei; Li, Hai; Chen, Yiran

doi:10.1109/nvmsa.2017.8064465

Cited by 29 publications

(16 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In terms of the accuracy-sensitive scenario, a rate-loss as large as 2.9% can be considered to be significant. However, some neural network’s applications such as Quantization Neural Network (QNN) may focus on the simple and fast computation more than the network’s recognition performance [32,33,34,35]. Usually, for the edge-computing applications, the power and time should be seriously considered.…”

Section: Resultsmentioning

confidence: 99%

“…One more thing to note here is that the rate loss due to the partial gating scheme can be compared to the state-of-the-art CMOS-implemented QNN, which was developed to replace the high-precision computation with the low-precision one [32]. Using the low-precision multiplication can result in 4~6X speed-up, in spite of roughly ~1–3% loss of recognition performance [32,33,34]. This ~1–3% loss is comparable to the rate loss that is due to the partial gating scheme, as shown in Figure 7 and Figure 8.…”

Section: Discussionmentioning

confidence: 99%

“…(2) The memristor crossbar implemented from the network’s pruning algorithm [18,19]. (3) The memristor crossbar implemented from QNN algorithm [32,33,34] (Memristor Binarized NN [24]). (4) The proposed partial gating scheme.…”

Section: Discussionmentioning

confidence: 99%

See 2 more Smart Citations

Partial-Gated Memristor Crossbar for Fast and Power-Efficient Defect-Tolerant Training

2019

View full text Add to dashboard Cite

A real memristor crossbar has defects, which should be considered during the retraining time after the pre-training of the crossbar. For retraining the crossbar with defects, memristors should be updated with the weights that are calculated by the back-propagation algorithm. Unfortunately, programming the memristors takes a very long time and consumes a large amount of power, because of the incremental behavior of memristor’s program-verify scheme for the fine-tuning of memristor’s conductance. To reduce the programming time and power, the partial gating scheme is proposed here to realize the partial training, where only some part of neurons are trained, which are more responsible in the recognition error. By retraining the part, rather than the entire crossbar, the programming time and power of memristor crossbar can be significantly reduced. The proposed scheme has been verified by CADENCE circuit simulation with the real memristor’s Verilog-A model. When compared to retraining the entire crossbar, the loss of recognition rate of the partial gating scheme has been estimated only as small as 2.5% and 2.9%, for the MNIST and CIFAR-10 datasets, respectively. However, the programming time and power can be saved by 86% and 89.5% than the 100% retraining, respectively.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Partial-Gated Memristor Crossbar for Fast and Power-Efficient Defect-Tolerant Training

2019

View full text Add to dashboard Cite

show abstract

“…Except for device engineering optimization, algorithm-level technique is also helpful to relax the requirement of the resistance state resolution. By modifying the regularization in training, the weight distribution can be tuned to fit the resistance values [93]. In this way, algorithm-level weights are tailored to enhance the accuracy of eNVM-based NCS.…”

Section: Precisionmentioning

confidence: 99%

Neuromorphic Computing Systems: From CMOS To Emerging Nonvolatile Memory

Yang

Qiao

Chen

2019

IPSJ Transactions on System LSI Design Methodology

Self Cite

View full text Add to dashboard Cite

The end of Moore's Law and von Neumann bottleneck motivate researchers to seek alternative architectures that can fulfill the increasing demand for computation resources which cannot be easily achieved by traditional computing paradigm. As one important practice, neuromorphic computing systems (NCS) are proposed to mimic biological behaviors of neurons and synapses, and accelerate computation of neural networks. Traditional CMOS-based implementation of NCS, however, are subject to large hardware cost required to precisely replicate the biological properties. In very recent decade, emerging nonvolatile memory (eNVM) was introduced to NCS design due to its high computing efficiency and integration density. Similar to the circuits built on other nanoscale devices, eNVM-based NCS also suffers from many reliability issues. In this paper, we give a short survey about CMOS-and eNVM-based NCS, including their basic implementations and training and inference schemes in various applications. We also discuss the design challenges of these NCS and introduce some techniques that can improve the reliability, precision, scalability, and security of the NCS. At the end, we provide our insights on the design trend and future challenges of the NCS.

show abstract

“…[ 14 , 15 , 16 ]. Memristors demonstrated experimentally in 2008 [ 17 ] are nonvolatile memories, where both binary and multi-level values can be stored [ 18 , 19 , 20 ]. For the architecture, memristor crossbars can be a built-in 3-dimensional multi-layer structure that seems very similar to the biological neuronal structure observed in the human brain [ 21 , 22 , 23 , 24 ].…”

Section: Introductionmentioning

confidence: 99%

Memristor-CMOS Hybrid Neuron Circuit with Nonideal-Effect Correction Related to Parasitic Resistance for Binary-Memristor-Crossbar Neural Networks

Nguyen

Min

2021

Micromachines

View full text Add to dashboard Cite

Voltages and currents in a memristor crossbar can be significantly affected due to nonideal effects such as parasitic source, line, and neuron resistance. These nonideal effects related to the parasitic resistance can cause the degradation of the neural network’s performance realized with the nonideal memristor crossbar. To avoid performance degradation due to the parasitic-resistance-related nonideal effects, adaptive training methods were proposed previously. However, the complicated training algorithm could add a heavy computational burden to the neural network hardware. Especially, the hardware and algorithmic burden can be more serious for edge intelligence applications such as Internet of Things (IoT) sensors. In this paper, a memristor-CMOS hybrid neuron circuit is proposed for compensating the parasitic-resistance-related nonideal effects during not the training phase but the inference one, where the complicated adaptive training is not needed. Moreover, unlike the previous linear correction method performed by the external hardware, the proposed correction circuit can be included in the memristor crossbar to minimize the power and hardware overheads for compensating the nonideal effects. The proposed correction circuit has been verified to be able to restore the degradation of source and output voltages in the nonideal crossbar. For the source voltage, the average percentage error of the uncompensated crossbar is as large as 36.7%. If the correction circuit is used, the percentage error in the source voltage can be reduced from 36.7% to 7.5%. For the output voltage, the average percentage error of the uncompensated crossbar is as large as 65.2%. The correction circuit can improve the percentage error in the output voltage from 65.2% to 8.6%. Almost the percentage error can be reduced to ~1/7 if the correction circuit is used. The nonideal memristor crossbar with the correction circuit has been tested for MNIST and CIFAR-10 datasets in this paper. For MNIST, the uncompensated and compensated crossbars indicate the recognition rate of 90.4% and 95.1%, respectively, compared to 95.5% of the ideal crossbar. For CIFAR-10, the nonideal crossbars without and with the nonideal-effect correction show the rate of 85.3% and 88.1%, respectively, compared to the ideal crossbar achieving the rate as large as 88.9%.

show abstract

A quantization-aware regularized learning method in multilevel memristor-based neuromorphic computing system

Cited by 29 publications

References 13 publications

Partial-Gated Memristor Crossbar for Fast and Power-Efficient Defect-Tolerant Training

Partial-Gated Memristor Crossbar for Fast and Power-Efficient Defect-Tolerant Training

Neuromorphic Computing Systems: From CMOS To Emerging Nonvolatile Memory

Memristor-CMOS Hybrid Neuron Circuit with Nonideal-Effect Correction Related to Parasitic Resistance for Binary-Memristor-Crossbar Neural Networks

Contact Info

Product

Resources

About