Entropy-SGD: biasing gradient descent into wide valleys*

Abstract

This paper proposes a new optimization algorithm called Entropy-SGD for training deep neural networks that is motivated by the local geometry of the energy landscape. Local extrema with low generalization error have a large proportion of almost-zero eigenvalues in the Hessian with very few positive or negative eigenvalues. We leverage upon this observation to construct a local-entropy-based objective function that favors well-generalizable solutions lying in large flat regions of the energy landscape, while avoiding poorly-generalizable solutions located in the sharp valleys. Conceptually, our algorithm resembles two nested loops of SGD where we use Langevin dynamics in the inner loop to compute the gradient of the local entropy before each update of the weights. We show that the new objective has a smoother energy landscape and show improved generalization over SGD using uniform stability, under certain assumptions. Our experiments on convolutional and recurrent networks demonstrate that Entropy-SGD compares favorably to state-of-the-art techniques in terms of generalization error and training time.

MS-Net: Multi-Site Network for Improving Prostate Segmentation With Heterogeneous MRI Data

“…Intuitively, our idea of MSKT is in line with recent studies on the robustness of high posterior entropy [40], [44]. In our MSKT, each auxiliary branch only aims to better learn the specific knowledge from a certain dataset.…”

“…While in MSKT, the universal network has to mimic the ground truth label and the predictions of multiple auxiliary branches simultaneously. Compared with conventional supervised learning, MSKT provides additional multi-site information to regularize the universal network and increases its posterior entropy [40], which helps the shared kernels to explore more robust representation among multiple datasets. Moreover, the multi-branch architecture in MSKT could also perform as a positive feature regularization to the universal encoder by jointly training the auxiliary branches and the universal network.…”

‘Place-cell’ emergence and learning of invariant data with restricted Boltzmann machines: breaking and dynamical restoration of continuous symmetries in the weight space

“…This prevents the dynamic from getting trapped in local maxima, which is crucial for non-convex optimization landscapes, and it also directs the dynamics toward minima with wider basins of attraction [23]. It has been argued that the later effect contributes in improving generalization performance [24][25][26]. Though the convergence rate of SGA has a slower asymptotic rate than ordinary gradient descent, it often does not matter in practice for finite data sets, as the performance on the test set usually does not improve anymore once the asymptotic regime is reached [27].…”

Natural representation of composite data with replicated autoencoders

“…Intuitively, one expects weights with a high local entropy to generalize well since they are more robust with respect to perturbations of the parameters and the data and therefore less likely to be an artifact of overfitting. In fact, such flat minima have already been found to have better generalization properties for some deep architectures [15]. We call the optimization procedure that finds such minima robust optimization.…”

“…Therefore we take special measures to avoid overfitting, which would decrease generalization performance and would constitute evidence that the current representation in the hidden units is not corresponding to the basic features in the data. We note that in neural networks, overfitting has been connected to sharp minima in the loss function [14,15,16]. To avoid such minima, we modify the optimization procedure to prefer weights that are in the vicinity of other weights that have low loss, which is a measure of the flatness of the loss landscape.…”