On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima

Keskar, Nitish Shirish; Mudigere, Dheevatsa; Nocedal, Jorge; Smelyanskiy, Mikhail; Tang, Ping

doi:10.48550/arxiv.1609.04836

Cited by 307 publications

(436 citation statements)

References 0 publications

Supporting

Mentioning

422

Contrasting

Unclassified

Order By: Relevance

“…Our result lends support to the belief that the noise introduced by Stochastic Gradient Descent (SGD) is superior to the isotropic noise, which has been widely observed [18,36,40,38]. As an illustrative example, we plot the generalization errors of SGD, SGLD with the isotropic noise and SGLD with the optimal noise in Figure 1, where their training curves behave almost the same (do not show here).…”

Section: Introductionsupporting

confidence: 76%

Optimizing Information-theoretical Generalization Bounds via Anisotropic Noise in SGLD

Wang¹,

Zhang²,

Zhang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Recently, the information-theoretical framework has been proven to be able to obtain non-vacuous generalization bounds for large models trained by Stochastic Gradient Langevin Dynamics (SGLD) with isotropic noise. In this paper, we optimize the information-theoretical generalization bound by manipulating the noise structure in SGLD. We prove that with constraint to guarantee low empirical risk, the optimal noise covariance is the square root of the expected gradient covariance if both the prior and the posterior are jointly optimized. This validates that the optimal noise is quite close to the empirical gradient covariance. Technically, we develop a new information-theoretical bound that enables such an optimization analysis. We then apply matrix analysis to derive the form of optimal noise covariance. Presented constraint and results are validated by the empirical observations.

show abstract

Section: Introductionsupporting

confidence: 76%

Optimizing Information-theoretical Generalization Bounds via Anisotropic Noise in SGLD

Wang¹,

Zhang²,

Zhang³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Analysis on the shape of the loss landscape For more explicit investigation, we analyze DCutMix with its loss landscape. Flatness of the loss landscape near local minima has been considered as one key signal for achieving better generalization of the model in various situations, by the number of previous studies (Keskar et al 2016;Pereyra et al 2017;Zhang et al 2018;Chaudhari et al 2019;Cha et al 2020). The general interpretation on the shape of the loss landscape is that if a model converges to a wide (flat) local minima, the model tends to have better generalization performance on unseen test dataset.…”

Section: Experimental Results On Image Classification Cifar-10/100mentioning

confidence: 99%

Observations on K-image Expansion of Image-Mixing Augmentation for Classification

Jeong¹,

Cha²,

Yoo³

et al. 2021

Preprint

View full text Add to dashboard Cite

Image-mixing augmentations (e.g., Mixup or CutMix), which typically mix two images, have become de-facto training tricks for image classification. Despite their huge success on image classification, the number of images to mix has not been profoundly investigated by the previous works, only showing the naive K-image expansion leads to poor performance degradation. This paper derives a new K-image mixing augmentation based on the stick-breaking process under Dirichlet prior. We show that our method can train more robust and generalized classifiers through extensive experiments and analysis on classification accuracy, a shape of a loss landscape and adversarial robustness, than the usual two-image methods. Furthermore, we show that our probabilistic model can measure the samplewise uncertainty and can boost the efficiency for Network Architecture Search (NAS) with 7x reduced search time.

show abstract

“…However, we argue that such linear connectivity can only be guaranteed when the loss surface around local minima is flat [8,17,18], which allows the low-error ellipsoid around each optimum to be wide enough to overlap with each other. However, when fine-tuning with extremely small data, a model is often suffering from large-batch training dilemma [19], which results in sharp and narrow minima. Thus the chance of existing loss-smooth linear connector between different tasks quickly becomes very small as the learning continues, illustrated as in Figure 2 (a).…”

Section: Stable Moment Matching (Smm)mentioning

confidence: 99%

Towards Generalized and Incremental Few-Shot Object Detection

Li¹,

Zhu²,

Ma³

et al. 2021

Preprint

View full text Add to dashboard Cite

Real-world object detection is highly desired to be equipped with the learning expandability that can enlarge its detection classes incrementally. Moreover, such learning from only few annotated training samples further adds the flexibility for the object detector, which is highly expected in many applications such as autonomous driving, robotics, etc. However, such sequential learning scenario with few-shot training samples generally causes catastrophic forgetting and dramatic overfitting. In this paper, to address the above incremental few-shot learning issues, a novel Incremental Few-Shot Object Detection (iFSOD) method is proposed to enable the effective continual learning from few-shot samples. Specifically, a Double-Branch Framework (DBF) is proposed to decouple the feature representation of base and novel (few-shot) class, which facilitates both the old-knowledge retention and new-class adaption simultaneously. Furthermore, a progressive model updating rule is carried out to preserve the long-term memory on old classes effectively when adapt to sequential new classes. Moreover, an inter-task class separation loss is proposed to extend the decision region of new-coming classes for better feature discrimination. We conduct experiments on both Pascal VOC and MS-COCO, which demonstrate that our method can effectively solve the problem of incremental few-shot detection and significantly improve the detection accuracy on both base and novel classes.

show abstract

On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima

Cited by 307 publications

References 0 publications

Optimizing Information-theoretical Generalization Bounds via Anisotropic Noise in SGLD

Optimizing Information-theoretical Generalization Bounds via Anisotropic Noise in SGLD

Observations on K-image Expansion of Image-Mixing Augmentation for Classification

Towards Generalized and Incremental Few-Shot Object Detection

Contact Info

Product

Resources

About