On the steady-state probability distribution of nonequilibrium stochastic systems

Noh, Jae Dong; Lee, Joongul

doi:10.3938/jkps.66.544

Cited by 3 publications

(1 citation statement)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We therefore split the analysis into two cases: Let us introduce these cases with an example from Noh and Lee (2015). the gradient ∇ f (x) , red dots denote the most likely locations of the steady-state e −Φ while the potential Φ is plotted as a contour map.…”

Section: Sgd For Deep Network Is Out-of-equilibriummentioning

confidence: 99%

Stochastic Gradient Descent Performs Variational Inference, Converges to Limit Cycles for Deep Networks

Chaudhari

Soatto

2018

2018 Information Theory and Applications Workshop (ITA)

171

237

View full text Add to dashboard Cite

Stochastic gradient descent (SGD) is widely believed to perform implicit regularization when used to train deep neural networks, but the precise manner in which this occurs has thus far been elusive. We prove that SGD minimizes an average potential over the posterior distribution of weights along with an entropic regularization term. This potential is however not the original loss function in general. So SGD does perform variational inference, but for a different loss than the one used to compute the gradients. Even more surprisingly, SGD does not even converge in the classical sense: we show that the most likely trajectories of SGD for deep networks do not behave like Brownian motion around critical points. Instead, they resemble closed loops with deterministic components. We prove that such "out-of-equilibrium" behavior is a consequence of highly non-isotropic gradient noise in SGD; the covariance matrix of mini-batch gradients for deep networks has a rank as small as 1% of its dimension. We provide extensive empirical validation of these claims, proven in the appendix.

show abstract

Section: Sgd For Deep Network Is Out-of-equilibriummentioning

confidence: 99%