A DNN Optimizer that Improves over AdaBelief by Suppression of the Adaptive Stepsize Range

Abstract

Adam and AdaBelief compute and make use of elementwise adaptive stepsizes in training deep neural networks (DNNs) by tracking the exponential moving average (EMA) of the squared-gradient g 2 t and the squared prediction error (mt −gt) 2 , respectively, where mt is the first momentum at iteration t and can be viewed as a prediction of gt. In this work, we attempt to find out if layerwise gradient statistics can be expoited in Adam and AdaBelief to allow for more effective training of DNNs. We address the above research question in two steps. Firstly, we slightly modify Adam and AdaBelief by introducing layerwise adaptive stepsizes in their update procedures via either pre or post processing. Empirical study indicates that the slight modification produces comparable performance for training VGG and ResNet models over CIFAR10, suggesting that layer-wise gradient statistics plays an important role towards the success of Adam and AdaBelief for at least certain DNN tasks. In the second step, instead of manual setup of layer-wise stepsizes, we propose Aida, a new optimisation method, with the objective that the elementwise stepsizes within each layer have significantly small statistic variances. Motivated by the fact that (mt − gt) 2 in AdaBelief is conservative in comparison to g 2 t in Adam in terms of layerwise statistic averages and variances, Aida is designed by tracking a more conservative function of mt and gt than (mt − gt) 2 in AdaBelief via layerwise orthogonal vector projections. Experimental results show that Aida produces either competitive or better performance with respect to a number of existing methods including Adam and AdaBelief for a set of challenging DNN tasks.