A local minimum for the 2-3-1 XOR network

Abstract

It was assumed proven that two-layer feedforward neural networks with t-1 hidden nodes, when presented with t input patterns, can not have any suboptimal local minima on the error surface. In this paper, however, we shall give a counterexample to this assumption. This counterexample consists of a region of local minima with nonzero error on the error surface of a neural network with three hidden nodes when presented with four patterns (the XOR problem). We will also show that the original proof is valid only when an unusual definition of local minimum is used.

Towards a Mathematical Understanding of the Difficulty in Learning with Feedforward Neural Networks

“…Results in [18,19] claim that, with the presence of dummy units, only T −1 units in the hidden layer are required to achieve exact learning and eliminate all suboptimal local minima. Such a statement has been shown to be false by counterexample utilising the XOR problem [16]. In the following proposition, we reinvestigate this problem as a concrete example of applying Proposition 3.…”

“…Firstly, we revisit some classic results on MLPs with only one hidden layer [18,19], and exemplify our analysis using the classic XOR problem [15,12,13,16]. For a learning task with T unique training samples, a finite exact approximator is realisable with a two-layer MLP (L = 2) having T units in the hidden layer, and its training process exempts from suboptimal local minima.…”

“…In the rest of this section, we examine these statements on the XOR problem [12,13,16]. Two specific MLP structures have been extensively studied in the literature, namely, the F (2, 2, 1) network and the F (2, 3, 1) network.…”

“…One major approach to address the problem of undesired local minima in MLP training is via an error/loss surface analysis [12,13,14]. Even for simple tasks, such as the classic XOR problem, the error surface analysis is surprisingly complicated and its results can be hard to conclude [15,13,16]. Early efforts in [17,18,19] try to identify general conditions on the topology of MLPs to eliminate undesired local minima, i.e., suboptimal local minima.…”

Using Upper Layer Weights to Efficiently Construct and Train Feedforward Neural Networks Executing Backpropagation

“…However, with only two middle nodes present, the network can get stuck in a local minimum while trying to navigate the error surface. When three nodes are present this is much less likely, which explains the increased performance from adding a third neuron to the middle layer (Sprinkhuizen-Kuyper & Boers, 1999). It appears that the MSSW performs as well as the other heuristics.…”