Neuro chips with on-chip backprop and/or Hebbian learning

Shima, Takeshi; Kimura, Taiki; Kamatani, Y.; Itakura, Tetsuro; Fujita, Yasuhiko; Iida, Takuya

doi:10.1109/isscc.1992.200450

Cited by 14 publications

(10 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It is possible to design analog circuits that approximate these characteristics but the result is a rather large synapse and thus an expensive solution. Two examples of backpropagation learning chips support this claim: Shima's [14] synapse consumes 1 250 K and Morie's [15] synapse consumes 525 K . These are much larger than the 4.9 K synapse that is described later in this paper.…”

Section: Learning Algorithms For Hardware Implementationsmentioning

confidence: 97%

See 1 more Smart Citation

Toward a general-purpose analog VLSI neural network with on-chip learning

Montalvo

Gyurcsik

Paulos

1997

IEEE Trans. Neural Netw.

View full text Add to dashboard Cite

This paper describes elements necessary for a general-purpose low-cost very large scale integration (VLSI) neural network. By choosing a learning algorithm that is tolerant of analog nonidealities, the promise of high-density analog VLSI is realized. A 64-synapse, 8-neuron proof-of-concept chip is described. The synapse, which occupies only 4900 mum(2) in a 2-mum technology, includes a hybrid of nonvolatile and dynamic weight storage that provides fast and accurate learning as well as reliable long-term storage with no refreshing. The architecture is user-configurable in any one-hidden-layer topology. The user-interface is fully microprocessor compatible. Learning is accomplished with minimal external support; the user need only present inputs, targets, and a clock. Learning is fast and reliable. The chip solves four-bit parity in an average of 680 ms and is successful in about 96% of the trials.

show abstract

Section: Learning Algorithms For Hardware Implementationsmentioning

confidence: 97%

“…Some implementations have consumed very low power [12] or have supported low power standby modes [10], [13]. Onchip learning has been demonstrated [14]- [19]. Some analog implementations have been very flexible [16], [20] and others have offered board-level alterability [21].…”

Section: Introductionmentioning

confidence: 99%

Toward a general-purpose analog VLSI neural network with on-chip learning

Montalvo

Gyurcsik

Paulos

1997

IEEE Trans. Neural Netw.

View full text Add to dashboard Cite

show abstract

“…This is not just an approximation of the gradient descent, but most of the time exhibits significantly different behavior, particularly when the learning rate is not small enough with respect to the number of training patterns [13], and learning appears slower when more weights reach the limits of the clipping function. During training other clipping methods were studied [14,15], for example on-chip training was first performed without quantization to get an initial estimate of the weights, then training was completed on a quantized network. In another approach, neural networks were trained off-chip and then weights were clipped for recall.…”

Section: The Effects Of Quantization Error On Backpropagationmentioning

confidence: 99%

Analysis of quantization effects on high-order function neural networks

Jiang

Gielen

2007

Appl Intell

View full text Add to dashboard Cite

In this paper we investigate the combined effects of quantization and clipping on high-order function neural networks (HOFNN). Statistical models are used to analyze the effects of quantization in a digital implementation. We analyze the performance degradation caused as a function of the number of fixed-point and floating-point quantization bits under the assumption of different probability distributions for the quantized variables, and then compare the training performance between situations with and without weight clipping. We establish and analyze the relationships for a true nonlinear neuron between inputs and outputs bit resolution, training and quantization methods, network order and performance degradation, all based on statistical models, and for on-chip and off-chip training. Our experimental simulation results verify the presented theoretical analysis.

show abstract

“…In this section, we will give two examples of the implementation of the algorithms for a feedforward network using and the squared error cost function. Thus, topologically this version of nonlinear backpropagation maps in exactly the same way on hardware as the original backpropagation algorithm-the only difference being how is calculated for backpropagation) [9] (see also [10]). …”

Section: Test Of Algorithmmentioning

confidence: 99%

Nonlinear backpropagation: doing backpropagation without derivatives of the activation function

Hertz

Krogh

Lautrup

et al. 1997

IEEE Trans. Neural Netw.

View full text Add to dashboard Cite

Abstract-The conventional linear backpropagation algorithm is replaced by a nonlinear version, which avoids the necessity for calculating the derivative of the activation function. This may be exploited in hardware realizations of neural processors. In this paper we derive the nonlinear backpropagation algorithms in the framework of recurrent backpropagation and present some numerical simulations of feedforward networks on the NetTalk problem. A discussion of implementation in analog very large scale integration (VLSI) electronics concludes the paper.

show abstract

Neuro chips with on-chip backprop and/or Hebbian learning

Cited by 14 publications

References 2 publications

Toward a general-purpose analog VLSI neural network with on-chip learning

Toward a general-purpose analog VLSI neural network with on-chip learning

Analysis of quantization effects on high-order function neural networks

Nonlinear backpropagation: doing backpropagation without derivatives of the activation function

Contact Info

Product

Resources

About