A simultaneous learning method for both activation functions and connection weights of multilayer neural networks

Abstract

This paper proposes a simultaneous learning algorithm for both activation functions and connection weights. The activation function is composed of several basic functions, such as sigmoidal function, Gaussian function and so on. In order to avoid local minima, the activation functions are controlled and randomly disturbed every some epochs. The activation functions are automatically optimized for each application. Probability and speed of learning are higher than the conventionals.

Wireless Network Monitoring System for River Ice Environment Based on a Data Fusion Algorithm

“…As the standard BP algorithm is based on the gradient descent method [23], the network training is carried out by calculating the gradient of the BP network weights and deviation to the objective function, the iterative process of the weight and t deviation can be expressed as:…”

A structure trainable neural network with embedded gating units and its learning algorithm

“…It is similar to the activation function training [6], [7]. Compared with the above, the hidden unit outputs are replaced by the gate unit outputs in the new structure.…”

“…Therefore, the update formula proposed for the trainable activation functions can be basically applied [6], [7].…”