Asymptotics of representation learning in finite Bayesian neural networks

Abstract

Recent works have suggested that finite Bayesian neural networks may outperform their infinite cousins because finite networks can flexibly adapt their internal representations. However, our theoretical understanding of how the learned hidden layer representations of finite networks differ from the fixed representations of infinite networks remains incomplete. Perturbative finite-width corrections to the network prior and posterior have been studied, but the asymptotics of learned features have not been fully characterized. Here, we argue that the leading finite-width corrections to the average feature kernels for any Bayesian network with linear readout and quadratic cost have a largely universal form. We illustrate this explicitly for two classes of fully connected networks: deep linear networks and networks with a single nonlinear hidden layer. Our results begin to elucidate which features of data wide Bayesian neural networks learn to represent.

The edge of chaos: quantum field theory and deep neural networks

“…We view our approaches as complementary, and warmly recommend [2] for readers interested in a pedagogical treatment of many ideas at this interface of theoretical physics and machine learning. We note that finite-width corrections to the feature kernel were also considered recently in [45].…”

“…In other words, in the phase space parametrized by (σ 2 w , σ 2 b ), the line σ2 w = γ 2 lies further to the right than the line σ 2 w,eff = γ 2 , past which the expression for the two-point function becomes complex. 45 Note that, as we discussed in subsec. 4.3.1, this does not imply that the true critical point does not exhibit a shift (and indeed, empirical observations demonstrate that there are sizable corrections to the result from the CLT; see, e.g., [17,18]), but simply that the critical point happens to lie at the edge of the strong coupling regime at which the present perturbative treatment breaks down.…”

“…As expected on general grounds, and explored quantitatively in [2,5,7], the Gaussian (N → ∞) limit does not suffice to describe real-world networks at finite width, whose distributions (i.e., actions) include effective interaction terms corresponding to higher cumulants. These finite-width effects are of both theoretical and practical relevance, and the depth-to-width ratio T /N was 45 As for the denominator, any zeros would require σ2 w = σ 2 w,eff , in which case the equation is invalid, since this would imply the existence of degenerate poles in the momentum space expression.…”

Contrasting random and learned features in deep Bayesian linear regression

“…As a result, a growing number of recent works have aimed to study the behavior of networks near the kernel limit, with the hope that leading-order corrections to the large-width behavior might elucidate how width and depth affect inference [4,[27][28][29][30][31][32][33][34][35][36][37][38][39]. Some of these works focus on the properties of the function-space prior distribution [27][28][29][30][31][32], some consider maximum-likelihood inference with gradient descent [30,38,39], and some consider properties of the full Bayes posterior [4,28,30,[33][34][35][36][37].…”

“…As a result, a growing number of recent works have aimed to study the behavior of networks near the kernel limit, with the hope that leading-order corrections to the large-width behavior might elucidate how width and depth affect inference [4,[27][28][29][30][31][32][33][34][35][36][37][38][39]. Some of these works focus on the properties of the function-space prior distribution [27][28][29][30][31][32], some consider maximum-likelihood inference with gradient descent [30,38,39], and some consider properties of the full Bayes posterior [4,28,30,[33][34][35][36][37]. This body of research has resulted in several conjectural conditions under which when narrower and deeper networks might perform better than their infinitely-wide cousins in the Bayesian setting, as measured by generalization for fixed data [33,35,36] or by some alternative criterion based on entropic considerations [30].…”