A Reconfigurable Two‐WSe₂‐Transistor Synaptic Cell for Reinforcement Learning

Abstract

learning, reinforcement learning (RL) is one of machine learning paradigms, which allow the machine to interact with the environment and update the policy according to the negative or positive reward signals. [10,11] The algorithms of RL [10] mainly include artificial neural network (ANN)-based deep Q-learning (DQN) and spiking neural network (SNN)based reward-modulated spike-timingdependent plasticity (R-STDP). Compared with ANN-based DQN that adopts error backpropagation and gradient descent to update the weight, R-STDP updates the weight by incorporating the brain-inspired STDP with reward signals being biologically more plausible. [12][13][14][15][16] The hardware implementation of RL paradigm with DQN based on emerging nonvolatile memory technologies (NVMs) has been reported recently. [17][18][19] However, hardware realization of R-STDP based on NVMs is far more less explored. The R-STDP learning rule mainly relies on the modulation of the "standard" STDP by a reward term (positive or negative reward), which is from the external environment, thus realizing SNN-based reinforcement learning. R-STDP stores the traces of synapses that are eligible for STDP (eligibility traces), and applies the modulated weight changes at the time of receiving a positive/negative reward signal. Eligibility traces are a critical ingredient of R-STDP learning rules. [20] There have been some works realizing long-lasting eligibility traces with complementary metal-oxide-semiconductor (CMOS) technology combining large capacitors. [21][22][23][24][25] Recently, Demirağ et al. exploit the drift behavior of phase change memory devices to intrinsically perform eligibility traces with long timescales realizing higher area efficiency than the CMOS ones. [26] In addition to realize eligibility traces constructing STDP, the polarity of STDP will be further changed according to the sign of external reward signal. Thus, it requires emerging nonvolatile devices with reconfigurable characteristics for both STDP and anti-STDP.2D semiconductor field-effect (FE) transistors (FETs) with atomic-level thickness show excellent electrostatic tunability, [27][28][29][30][31][32][33] which provides the possibility of modulating the channel to be reconfigurable p-type or n-type for both STDP and anti-STDP learning rules. On the other hand, to realize nonvolatile memory characteristics, we adopt the ferroelectric poly(vinylidene fluoride-trifluoroethylene) (P(VDF-TrFE)) as gate dielectric for dynamically tunable memory Reward-modulated spike-timing-dependent plasticity (R-STDP) is a braininspired reinforcement learning (RL) rule, exhibiting potential for decisionmaking tasks and artificial general intelligence. However, the hardware implementation of the reward-modulation process in R-STDP usually requires complicated Si complementary metal-oxide-semiconductor (CMOS) circuit design that causes high power consumption and large footprint. Here, a design with two synaptic transistors (2T) connected in a parallel structure is experimentally demonstrated. The 2T...

Ferroelectric Polarized in Transistor Channel Polarity Modulation for Reward-Modulated Spike-Time-Dependent Plasticity Application

“…In order to assess the weight of synaptic renewal, the nonlinearity is extracted. 51 The nonlinearity of n/p-type channels was (1.22, −1.37)/(1.89, −2.63) as shown in Figure 2(e,g), manifesting that our carbon nanotube 2T-based synapse had good linearity. In addition, under the excitation of 50 consecutive same pulses (±7 V amplitude, 6 μs duration), the ON/OFF ratio of n/p-channels reached 55 (22/0.4 μS).…”

“…Figure (d,f) describes the weight updating characteristics of the p-channel and n-channel under 50 sequential programming signals of ±7 V amplitude, 6 μs duration. In order to assess the weight of synaptic renewal, the nonlinearity is extracted . The nonlinearity of n/p-type channels was (1.22, −1.37)/(1.89, −2.63) as shown in Figure (e,g), manifesting that our carbon nanotube 2T-based synapse had good linearity.…”

“…A schematic diagram of the R-STDP mode was exhibited in Figure (a), and the eligibility trace filtered STDP learning window was multiplied by the reward signal to produce modulation and even polarity changes of the STDP. The weight changes with time: d w /d t = c ( t )­d­( t ), where c ( t ) denotes the eligibility trace storing the temporary memory of STDP and d­( t ) is the reward signal. So, rewards cause STDP, and punishments cause anti-STDP.…”

2D semiconductors for specific electronic applications: from device to system

“…The memory of 2D semiconductors is often combined with computational processing, since the stored state can finely regulate the conductance of the 2D semiconductor, and then act with the input signal to obtain the processed output. This is similar to applications in neuromorphic computing, where there has been a large amount of research linking device behaviour to the mapping of neuro-synaptic behaviour, but for 2D semiconductors is mostly limited to functional demonstrations of single devices [162][163][164] . Considering the variation between multiple devices and achieving integration at the array level is critical to implementing large-scale systems that highlight the advantages of 2D.…”

Ferroelectrics-Integrated Two-Dimensional Devices toward Next-Generation Electronics

“…Long-term potentiation and depression were also achieved in this device (Figure e), and the synaptic weight change varied with the timing difference between incoming pre- and postsynaptic spikes, resulting in spike-timing-dependent plasticity (STDP) that was realized by the carefully designed voltage waveform (Figure f) . Similar FeFETs have also been implemented in artificial neural networks for neuromorphic applications including associative learning, reinforcement learning, and recognition tasks. − …”