Gradient Noise Convolution (GNC): Smoothing Loss Function for Distributed Large-Batch SGD

Haruki, Kosuke; Suzuki, Taiji; Hamakawa, Yohei; Toda, Takeshi; Sakai, Ryuji; Ozawa, Masashi; Kimura, Mitsuhiro

doi:10.48550/arxiv.1906.10822

Cited by 3 publications

(4 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In our paper, we show that parameter averaging stabilizes the convergence to a flat region or asymmetric valley, and suggest the usefulness of the combination with the large step size for the difficult dataset which needs a stronger regularization. Besides, several authors proposed methods that explicitly inject noise for improving generalization (Chaudhari et al, 2019), in particular for the large batch setting (Wen et al, 2018;Haruki et al, 2019;Lin et al, 2020). 2018) explained that SGD travels on the hypersphere because of the convergence to Gaussian distribution and the concentration on the sphere under a simplified setting, and thus averaging scheme allows us to go inside of the sphere which may be flat.…”

Section: Discussionmentioning

confidence: 99%

Parameter Averaging for SGD Stabilizes the Implicit Bias towards Flat Regions

Nitanda¹,

Kikuchi²,

Maeda³

2023

Preprint

View full text Add to dashboard Cite

Stochastic gradient descent is a workhorse for training deep neural networks due to its excellent generalization performance. Several studies demonstrated this success is attributed to the implicit bias of the method that prefers a flat minimum and developed new methods based on this perspective. Recently, Izmailov et al. (2018) empirically observed that an averaged stochastic gradient descent with a large step size can bring out the implicit bias more effectively and can converge more stably to a flat minimum than the vanilla stochastic gradient descent. In our work, we theoretically justify this observation by showing that the averaging scheme improves the bias-optimization tradeoff coming from the stochastic gradient noise: a large step size amplifies the bias but makes convergence unstable, and vice versa. Specifically, we show that the averaged stochastic gradient descent can get closer to a solution of a penalized objective on the sharpness than the vanilla stochastic gradient descent using the same step size under certain conditions. In experiments, we verify our theory and show this learning scheme significantly improves performance.

show abstract

Section: Discussionmentioning

confidence: 99%

Parameter Averaging for SGD Stabilizes the Implicit Bias towards Flat Regions

Nitanda¹,

Kikuchi²,

Maeda³

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…To fairly measure essential performance gains by our method, we reproduced recent other SOAT optimization methods which include K-FAC 30 and GNC. 31 The baseline we used in these cases is not applied any optimizations, both instances use a multi-steps learning rate policy. In distributed training scenarios, the ImageNet baseline results include floating-point randomness, due to performing allreduce, which is a collective communication operation among multi-nodes.…”

Section: Compare With Other Optimization Techniquesmentioning

confidence: 99%

“…To overcome the generalization performance degrading in the large-bath training, Wen 32 ad Haruki 31 proposed gradient noise injecting methods to eliminate sharp minima by adding empirically gradient noise to weight, because small-batch SGD introduces noisy behavior that allows escaping from sharp minima. 16 Haruki claimed that their method can achieve good results on ResNet-50 training ImageNet-1K with batch size 32 K, but not on 128 K. Moreover, due to the randomness of gradient noise, it could not achieve a good performance every time.…”

Section: Related Workmentioning

confidence: 99%

Fast and accurate variable batch size convolution neural network training on large scale distributed systems

Xiao

Sun

et al. 2022

Concurrency and Computation

View full text Add to dashboard Cite

Large-scale distributed convolution neural network (CNN) training brings two performance challenges: model performance and system performance. Large batch size usually leads to model test accuracy loss, which counteracts the benefits of parallel SGD. The existing solutions require massive hyperparameter hand-tuning. To overcome this difficult, we analyze the training process and find that earlier training stages are more sensitive to batch size. Accordingly, we assert that different stages should use different batch size, and propose a variable batch size strategy. In order to remain high test accuracy under larger batch size cases, we design an auto-tuning engine for automatic parameter tuning in the proposed variable batch size strategy. Furthermore, we develop a dataflow implementation approach to achieve the high-throughput CNN training on supercomputer system. Our approach has achieved high generalization performance on SOAT CNN networks. For the ShuffleNet, training with ImageNet-1K dataset, we scale the batch size to 120 K without accuracy loss and to 128 K with only a slight loss. And the dataflow implementation approach achieves 93.5% scaling efficiency on 1024 GPUs compared with the state-of-the-art.

show abstract

“…Lookahead (Zhang et al, 2019) as of our interest here in spirit is closer to extrapolation methods (Korpelevich, 1976) which rely on gradients taken not at the current iterate but at an extrapolated point for the current trajectory. For highly complex optimization landscapes such as in deep learning, the effect of using gradients at perturbations of the current iterate has a desirable smoothing effect which is known to help training speed and stability in the case of non-convex single-objective optimization (Wen et al, 2018;Haruki et al, 2019) GANs. Several proposed methods for GANs are motivated by the "recurrent dynamics".…”

Section: Related Workmentioning

confidence: 99%

Taming GANs with Lookahead-Minmax

Chavdarova¹,

Pagliardini²,

Stich³

et al. 2020

Preprint

View full text Add to dashboard Cite

Generative Adversarial Networks are notoriously challenging to train. The underlying minimax optimization is highly susceptible to the variance of the stochastic gradient and the rotational component of the associated game vector field. We empirically demonstrate the effectiveness of the Lookahead meta-optimization method for optimizing games, originally proposed for standard minimization. The backtracking step of Lookahead naturally handles the rotational game dynamics, which in turn enables the gradient ascent descent method to converge on challenging toy games often analyzed in the literature. Moreover, it implicitly handles high variance without using large mini-batches, known to be essential for reaching state of the art performance. Experimental results on MNIST, SVHN, and CIFAR-10, demonstrate a clear advantage of combining Lookahead with Adam or extragradient, in terms of performance, memory footprint, and improved stability. Using 30-fold fewer parameters and 16-fold smaller minibatches we outperform the reported performance of the class-dependent BigGAN on CIFAR-10 by obtaining FID of 13.65 without using the class labels, bringing state-of-the-art GAN training within reach of common computational resources.

show abstract

Gradient Noise Convolution (GNC): Smoothing Loss Function for Distributed Large-Batch SGD

Cited by 3 publications

References 0 publications

Parameter Averaging for SGD Stabilizes the Implicit Bias towards Flat Regions

Parameter Averaging for SGD Stabilizes the Implicit Bias towards Flat Regions

Fast and accurate variable batch size convolution neural network training on large scale distributed systems

Taming GANs with Lookahead-Minmax

Contact Info

Product

Resources

About