Efficient Training of the Memristive Deep Belief Net Immune to Non‐Idealities of the Synaptic Devices

Abstract

The tunability of conductance states of various emerging non-volatile memristive devices emulates the plasticity of biological synapses, making it promising in the hardware realization of large-scale neuromorphic systems. The inference of the neural network can be greatly accelerated by the vector-matrix multiplication (VMM) performed within a crossbar array of memristive devices in one step. Nevertheless, the implementation of the VMM needs complex peripheral circuits and the complexity further increases since non-idealities of memristive devices prevent precise conductance tuning (especially for the online training) and largely degrade the performance of the deep neural networks (DNNs). Here, we present an efficient online training method of the memristive deep belief net (DBN). The proposed memristive DBN uses stochastically binarized activations, reducing the complexity of peripheral circuits, and uses the contrastive divergence (CD) based gradient descent learning algorithm. The analog VMM and digital CD are performed separately in a mixed-signal hardware arrangement, making the memristive DBN high immune to non-idealities of synaptic devices. The number of write operations on memristive devices is reduced by two orders of magnitude. The recognition accuracy of 95%~97% can be achieved for the MNIST dataset using pulsed synaptic behaviors of various memristive synaptic devices.

Self-Powered Organic Optoelectronic Synapses with Binarized Weights for Noise-Suppressed Visual Perception and High-Robustness Inference

“…The results show a large A p of 5, and the number of effective conductance states in our organic synapses is 20 (Figure a). Device nonlinearity and limited conductance states are common problems among many neuromorphic synaptic devices, increasing the difficulty of device deployments and neural network training. − ,, Note that the fabricated organic synapses only worked in the conductance potentiation regime as light pulses were continuously applied, leading to positive weight values. Hence, the simulated synaptic weight in the hardware-based devices is derived from the difference between the conductance values of two equivalent organic synapses ( w = G + – G – , where G + and G – are the conductance values of two equivalent devices), realizing the negative weight update (inset in Figure b, Figure S9, and Note S9).…”

Binary‐Stochasticity‐Enabled Highly Efficient Neuromorphic Deep Learning Achieves Better‐than‐Software Accuracy

“…Limitation of Previous Utilizations of the Hardware Stochasticity: The BS that could be provided by the intrinsic noise or stochastic nature made them ready to be employed in Hopfield-type neural networks, such as finding the global minima in constraint satisfaction problems [23,[48][49][50] or learning through contrastive divergence in restricted Boltzmann machines. [51,52] However, for deep learning with the error backpropagation and gradient descent rule, the jigsaw puzzle of the in situ learning independent of the von-Neumann architecture within a neuromorphic system was not completed. For instance, a neural sampling machine with stochastic synapses of ferroelectric field effect transistors was recently reported for the learning and inference of a fully connected neural network.…”

“…An empirical model capturing the synaptic behavior of LTP and LTD under identical pulses was used to simulate the synaptic plasticity of the analog memristors with nonidealities. [ 52 ] This model considered the on/off ratio, the nonlinearities ( and ), the asymmetry between potentiation and depression, and the write variations. The median conductance changes (without cycle‐to‐cycle write variations) for a memristor device with conductance under potentiation pulses and depression pulses can be written as…”

“…First, the calculation of the gradient for an array of weights, a vector–vector outer product, could be performed parallelly. [ 52 ] Second, the gradient was accumulated in a unit step.…”

Ambient Stable All Inorganic CsCu2I3 Artificial Synapses for Neurocomputing

“…The calculated nonlinearity values (β P /β D ) are 1.8/1.3 for potentiation and depression, respectively. Supplementary Note 3 in the Supporting Information provides detailed weight update relationships with nonlinearity factors that have a significant impact on the neuromorphic computing performance. , As described in Figure d, each electrical stimulus induces the migration of iodine vacancies, which is displayed by the gradual change of the conductance of the device. When a negative voltage is applied to the top electrode, iodine vacancies with positive charge migrate and accumulate at the interface between the top electrode and the CsCu 2 I 3 thin film, forming a conductive filament.…”