Stability for the training of deep neural networks and other classifiers

Abstract

We examine the stability of loss-minimizing training processes that are used for deep neural networks (DNN) and other classifiers. While a classifier is optimized during training through a so-called loss function, the performance of classifiers is usually evaluated by some measure of accuracy, such as the overall accuracy which quantifies the proportion of objects that are well classified. This leads to the guiding question of stability: does decreasing loss through training always result in increased accuracy? We formalize the notion of stability, and provide examples of instability. Our main result consists of two novel conditions on the classifier which, if either is satisfied, ensure stability of training, that is we derive tight bounds on accuracy as loss decreases. We also derive a sufficient condition for stability on the training set alone, identifying flat portions of the data manifold as potential sources of instability. The latter condition is explicitly verifiable on the training dataset. Our results do not depend on the algorithm used for training, as long as loss decreases with training.

Stability of Accuracy for the Training of DNNs Via the Uniform Doubling Condition

“…The main goal of this paper is to provide conditions on the training set T which ensure that this does not happen, and so accuracy is bounded from below as the loss decreases. The results in this paper build on similar stability of accuracy results for classifiers given in [BJS21].…”

“…It is useful to review the first results given in this direction. In [BJS21] it was shown that small clusters in δX lead to instability of accuracy, and so a doubling condition on δX was introduced which prevents the existence of small clusters.…”

“…In [BJS21] a doubling condition was also introduced on the training set directly and it was shown that this doubling condition propagates through the DNN, such that if the training set satisfies the doubling condition than δX also satisfies the doubling condition. One can then ensure that δX satisfies the doubling condition by checking that the training set satisfies the doubling condition.…”

“…An example for when accuracy decreases even though loss decreases is given in Fig. 1 (taken from [BJS21]). Although training has progressed and loss has decreased, the accuracy has dropped drastically (from 95% to 79.8%).…”

Effects of Core Stability Training on Specific Strength in Wrestlers

“…The test indicators before and after the experiment were tested by paired T test, and the data results were expressed as mean ± standard deviation (x ± SD), p>0.05 means that the difference is not significant, 0.01<P≤0.05 means the difference is significant, and P≤0.01 means the difference is highly significant. 7,8…”