An iterative method for training multilayer networks with threshold functions

Abstract

Concerns the problem of finding weights for feed-forward networks in which threshold functions replace the more common logistic node output function. The advantage of such weights is that the complexity of the hardware implementation of such networks is greatly reduced. If the task to be learned does not change over time, it may be sufficient to find the correct weights for a threshold function network off-line and to transfer these weights to the hardware implementation. This paper provides a mathematical foundation for training a network with standard logistic function nodes and gradually altering the function to allow a mapping to a threshold unit network. The procedure is analogous to taking the limit of the logistic function as the gain parameter goes to infinity. It is demonstrated that, if the error in a trained network is small, a small change in the gain parameter will cause a small change in the network error. The result is that a network that must be implemented with threshold functions can first be trained using a traditional back propagation network using gradient descent, and further trained with progressively steeper logistic functions. In theory, this process could require many repetitions. In simulations, however, the weights have be successfully mapped to a true threshold network after a modest number of slope changes. It is important to emphasize that this method is only applicable to situations for which off-line learning is appropriate.

The Interchangeability of Learning Rate and Gain in Backpropagation Neural Networks

“…Approaching hard limiting thresholds by increasing the gain of the activation functions (Corwin et al 1994Yu et al 1994) is similar to multipling the weights with a constant greater than one. In the nal stage of the training process the activation functions can be replaced by a threshold if this does not cause a degradation in performance.…”

Training neural networks with threshold activation functions and constrained integer weights

“…not to change over time, in order to train the network "off-line" in a software simulation and later transfer it to the hardware [1,4]. But many real-life applications may not be static, i.e.…”

Training multilayer networks with discrete activation functions

“…In this section, we present comparative results for the DE algorithm, and the algorithms proposed in [4], [14], [3], [6], which are denoted in the tables below as (GLO), (T), (GZ), and (MVGA), respectively.…”

“…In [4], Corwin suggested to train NDAs with progressively steeper analog functions to facilitate training. Thus, in his experiments he used values such as β ∈ {2, 3, 5, 10} to alter the shape of the sigmoid from time to time during training.…”

“…Various modifications of the gradient descent have been presented to train NDAs [1,3,4,6,14,17]. However, these methods require to a certain degree, depending on the method, that the learning task should be static.…”