Sharpness-Aware Training for Free

Abstract

Modern deep neural networks (DNNs) have achieved state-of-the-art performances but are typically over-parameterized. The over-parameterization may result in undesirably large generalization error in the absence of other customized training strategies. Recently, a line of research under the name of Sharpness-Aware Minimization (SAM) has shown that minimizing a sharpness measure, which reflects the geometry of the loss landscape, can significantly reduce the generalization error. However, SAM-like methods incur a two-fold computational overhead of the given base optimizer (e.g. SGD) for approximating the sharpness measure. In this paper, we propose Sharpness-Aware Training for Free, or SAF, which mitigates the sharp landscape at almost zero additional computational cost over the base optimizer. Intuitively, SAF achieves this by avoiding sudden drops in the loss in the sharp local minima throughout the trajectory of the updates of the weights. Specifically, we suggest a novel trajectory loss, based on the KL-divergence between the outputs of DNNs with the current weights and past weights, as a replacement of the SAM's sharpness measure. This loss captures the rate of change of the training loss along the model's update trajectory. By minimizing it, SAF ensures the convergence to a flat minimum with improved generalization capabilities. Extensive empirical results show that SAF minimizes the sharpness in the same way that SAM does, yielding better results on the ImageNet dataset with essentially the same computational cost as the base optimizer. The code will be released.Preprint. Under review.

Gradient Norm Aware Minimization Seeks First-Order Flatness and Improves Generalization

“…And there also have been several simple strategies to achieve a smaller maximum Hessian eigenvalue, such as choosing a large learning rate [12,33,46] and smaller batch size [32,46,62]. Sharpness-Aware Minimization (SAM) [20] and its variants [17,18,39,44,49,54,83,86] are representative training algorithm to seek flat minima for better generalization. However, their definition of flatness is limited to zeroth-order flatness.…”

“…Especially, some works discuss the connection between the geometry of the loss landscape and generalization [20,23,31]. A branch of effective approaches, sharpness-Aware Minimization (SAM) [20] and its variants [17,18,39,49,54,83], minimizes the worst-case loss within a perturbation radius, which we call zeroth-order flatness. It is proven that optimizing the zeroth-order flatness leads to lower generalization error and achieves state-of-the-art performance on various image classification tasks [20,44,86].…”

mSAM: Micro-Batch-Averaged Sharpness-Aware Minimization

“…This proxy of sharpness lends itself to easy computation, unlike the measures of sharpness described earlier. SAM has sparked interest in sharpness-aware training, resulting in several variants [24,25,26,27,28].…”

Sharpness-Aware Minimization: An Implicit Regularization Perspective

“…Thus, in SAM, the loss function is modified, in a way that it encourages convergence to flatter regions of the loss. SAM has been shown to be empirically successful in numerous tasks (Bahri et al, 2021;Behdin et al, 2022;Chen et al, 2021) and has been extended to several variations (Du et al, 2022;Zhuang et al, 2022). Thus, there has been a growing interest in understanding the theoretical underpinnings of SAM.…”