Depth induces scale-averaging in overparameterized linear Bayesian neural networks

Abstract

Inference in deep Bayesian neural networks is only fully understood in the infinite-width limit, where the posterior flexibility afforded by increased depth washes out and the posterior predictive collapses to a shallow Gaussian process. Here, we interpret finite deep linear Bayesian neural networks as datadependent scale mixtures of Gaussian process predictors across output channels. We leverage this observation to study representation learning in these networks, allowing us to connect limiting results obtained in previous studies within a unified framework. In total, these results advance our analytical understanding of how depth affects inference in a simple class of Bayesian neural networks.

Contrasting random and learned features in deep Bayesian linear regression

“…Using the replica trick [45,46], we compute learning curves for simple linear regression, deep linear Gaussian random feature (RF) models, and deep linear NNs. Our results are obtained using an isotropic Gaussian likelihood in the limit of small likelihood variance, which renders this analysis analytically tractable [33,34]. Using alternative replica-free methods and numerical simulation, we show that the predictions obtained under a replica-symmetric (RS) Ansatz are accurate for all three model classes.…”

“…In the noise-free case, this limiting likelihood is matched to the true generative model of the data, but it is clearly mismatched in the presence of label noise. This limit has been considered in several recent studies of deep linear Bayesian neural networks [4,30,[33][34][35].…”

“…In Appendix D, we provide a detailed discussion of the mapping between the polynomial condition in terms of which their result is expressed and the RS condition (28). In Appendix D, we also use a finite-size fixed-data approach derived from our previous work [34] to show that the learning curve should be of the form (27). Concretely, this approach gives an expression for z as the thermodynamic limit of a dataset average of a ratio of prior averages, with the remaining components of the learning curve exactly matching the RS prediction.…”

“…As a result, a growing number of recent works have aimed to study the behavior of networks near the kernel limit, with the hope that leading-order corrections to the large-width behavior might elucidate how width and depth affect inference [4,[27][28][29][30][31][32][33][34][35][36][37][38][39]. Some of these works focus on the properties of the function-space prior distribution [27][28][29][30][31][32], some consider maximum-likelihood inference with gradient descent [30,38,39], and some consider properties of the full Bayes posterior [4,28,30,[33][34][35][36][37].…”

“…As a result, a growing number of recent works have aimed to study the behavior of networks near the kernel limit, with the hope that leading-order corrections to the large-width behavior might elucidate how width and depth affect inference [4,[27][28][29][30][31][32][33][34][35][36][37][38][39]. Some of these works focus on the properties of the function-space prior distribution [27][28][29][30][31][32], some consider maximum-likelihood inference with gradient descent [30,38,39], and some consider properties of the full Bayes posterior [4,28,30,[33][34][35][36][37]. This body of research has resulted in several conjectural conditions under which when narrower and deeper networks might perform better than their infinitely-wide cousins in the Bayesian setting, as measured by generalization for fixed data [33,35,36] or by some alternative criterion based on entropic considerations [30].…”