Neural Networks with Comparatively Few Critical Points

Abstract

A critical point is a point on which the derivatives of an error function are all zero. It has been shown in the literatures that the critical points caused by the hierarchical structure of the real-valued neural network could be local minima or saddle points, whereas most of the critical points caused by the hierarchical structure are saddle points in the case of complexvalued neural networks. Several studies have demonstrated that that kind of singularity has a negative effect on learning dynamics in neural networks. In this paper, we will demonstrate via some examples that the decomposition of highdimensional NNs into real-valued NNs equivalent to the original NNs yields the NNs that do not have critical points based on the hierarchical structure.