On-Chip Error-Triggered Learning of Multi-Layer Memristive Spiking Neural Networks

Payvand, Melika; Fouda, Mohammed E.; Kurdahi, Fadi; Eltawil, Ahmed M.; Neftci, Emre

doi:10.1109/jetcas.2020.3040248

Cited by 30 publications

(25 citation statements)

References 63 publications

(88 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For spiking neural networks, this property can prove important since gradient based methods have recently taken on renewed popularity in the training of such networks, especially through the use of surrogate gradient methods ( Neftci et al, 2019 ). An increasingly common practice, despite the lack of biological plausibility, is to use mini-batch GPU acceleration of spiking networks to train them more rapidly ( Neftci et al, 2017 ; Payvand et al, 2020 ). While researchers cite that future hardware will be able to more efficiently train using batch sizes of 1 ( Stewart et al, 2020 ), this has also frequently been proposed as the ideal batch size for using memristor-based artificial neural networks due to the memory overhead associated with gradient data.…”

Section: Resultsmentioning

confidence: 99%

Gradient Decomposition Methods for Training Neural Networks With Non-ideal Synaptic Devices

et al. 2021

View full text Add to dashboard Cite

While promising for high-capacity machine learning accelerators, memristor devices have non-idealities that prevent software-equivalent accuracies when used for online training. This work uses a combination of Mini-Batch Gradient Descent (MBGD) to average gradients, stochastic rounding to avoid vanishing weight updates, and decomposition methods to keep the memory overhead low during mini-batch training. Since the weight update has to be transferred to the memristor matrices efficiently, we also investigate the impact of reconstructing the gradient matrixes both internally (rank-seq) and externally (rank-sum) to the memristor array. Our results show that streaming batch principal component analysis (streaming batch PCA) and non-negative matrix factorization (NMF) decomposition algorithms can achieve near MBGD accuracy in a memristor-based multi-layer perceptron trained on the MNIST (Modified National Institute of Standards and Technology) database with only 3 to 10 ranks at significant memory savings. Moreover, NMF rank-seq outperforms streaming batch PCA rank-seq at low-ranks making it more suitable for hardware implementation in future memristor-based accelerators.

show abstract

Section: Resultsmentioning

confidence: 99%

Gradient Decomposition Methods for Training Neural Networks With Non-ideal Synaptic Devices

et al. 2021

View full text Add to dashboard Cite

show abstract

“…The implementation of these two operations in memristive array will further improve the performance of the deep learning accelerators, while Hebbian-based learning algorithms could potentially bypass these operations. Online versions of Backprop, as discussed in section 3, are very recent and a memristive-based hardware demonstration is not yet available, despite some work in this direction is being done (Payvand et al, 2020b). To implement adaptation, biologically plausible algorithms able to cope with the non-ideal characteristics of memristive devices are needed.…”

Section: Applications Of Memristive Neural Networkmentioning

confidence: 99%

“…Online versions of Backprop, as discussed in section 3, are very recent and a memristive-based hardware demonstration is not yet available, despite some work in this direction is being done (Payvand et al, 2020b ). To implement adaptation, biologically plausible algorithms able to cope with the non-ideal characteristics of memristive devices are needed.…”

Section: Memristive Devices and Computingmentioning

confidence: 99%

Adaptive Extreme Edge Computing for Wearable Devices

et al. 2021

Self Cite

View full text Add to dashboard Cite

Wearable devices are a fast-growing technology with impact on personal healthcare for both society and economy. Due to the widespread of sensors in pervasive and distributed networks, power consumption, processing speed, and system adaptation are vital in future smart wearable devices. The visioning and forecasting of how to bring computation to the edge in smart sensors have already begun, with an aspiration to provide adaptive extreme edge computing. Here, we provide a holistic view of hardware and theoretical solutions toward smart wearable devices that can provide guidance to research in this pervasive computing era. We propose various solutions for biologically plausible models for continual learning in neuromorphic computing technologies for wearable sensors. To envision this concept, we provide a systematic outline in which prospective low power and low latency scenarios of wearable sensors in neuromorphic platforms are expected. We successively describe vital potential landscapes of neuromorphic processors exploiting complementary metal-oxide semiconductors (CMOS) and emerging memory technologies (e.g., memristive devices). Furthermore, we evaluate the requirements for edge computing within wearable devices in terms of footprint, power consumption, latency, and data size. We additionally investigate the challenges beyond neuromorphic computing hardware, algorithms and devices that could impede enhancement of adaptive edge computing in smart wearable devices.

show abstract

“…Considering that in most practical applications, ANNs show very good performance, several works focus on the conversion of ANNs to SNNs [ 30 ]. Also, high performance deep SNNs were implemented with several learning methods [ 41 , 42 , 43 ] including gradient descent [ 44 , 45 ]. Other learning methods were developed for the detection of spatio-temporal patterns [ 46 , 47 ] and for evolving SNN [ 48 ].…”

Section: Related Workmentioning

confidence: 99%

Adaptive SNN for Anthropomorphic Finger Control

Hulea

Uleru

Caruntu

2021

Sensors

View full text Add to dashboard Cite

Anthropomorphic hands that mimic the smoothness of human hand motions should be controlled by artificial units of high biological plausibility. Adaptability is among the characteristics of such control units, which provides the anthropomorphic hand with the ability to learn motions. This paper presents a simple structure of an adaptive spiking neural network implemented in analogue hardware that can be trained using Hebbian learning mechanisms to rotate the metacarpophalangeal joint of a robotic finger towards targeted angle intervals. Being bioinspired, the spiking neural network drives actuators made of shape memory alloy and receives feedback from neuromorphic sensors that convert the joint rotation angle and compression force into the spiking frequency. The adaptive SNN activates independent neural paths that correspond to angle intervals and learns in which of these intervals the rotation the finger rotation is stopped by an external force. Learning occurs when angle-specific neural paths are stimulated concurrently with the supraliminar stimulus that activates all the neurons that inhibit the SNN output stopping the finger. The results showed that after learning, the finger stopped in the angle interval in which the angle-specific neural path was active, without the activation of the supraliminar stimulus. The proposed concept can be used to implement control units for anthropomorphic robots that are able to learn motions unsupervised, based on principles of high biological plausibility.

show abstract

On-Chip Error-Triggered Learning of Multi-Layer Memristive Spiking Neural Networks

Cited by 30 publications

References 63 publications

Gradient Decomposition Methods for Training Neural Networks With Non-ideal Synaptic Devices

Gradient Decomposition Methods for Training Neural Networks With Non-ideal Synaptic Devices

Adaptive Extreme Edge Computing for Wearable Devices

Adaptive SNN for Anthropomorphic Finger Control

Contact Info

Product

Resources

About