Bayesian Deep Ensembles via the Neural Tangent Kernel

He, Bobby; Lakshminarayanan, Balaji; Teh, Yee Whye

doi:10.48550/arxiv.2007.05864

Cited by 14 publications

(23 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…where γ n → ∞ is a sequence with arbitrarily slow growth, we can verify that there exists an M > 0 such that both (40) and (43) converges to zero by noting that as n → ∞, mτ 2 m → ∞, τ 2 m → 0. Hence, the term (I) converges to zero.…”

Section: B2 Proof Of Theorem 31mentioning

confidence: 99%

“…While the algorithm can be directly applied to neural network models as in [23], we follow [40] and modify the objective, to account for the difference between the neural tangent kernel (NTK) [41] of a wide neural network architecture and the NNGP kernel of the corresponding infinite-width Bayesian neural network [42][43][44]. Concretely, we modify (13) as…”

Section: Scalable Approximate Inference Via a Randomized Prior Trickmentioning

confidence: 99%

See 1 more Smart Citation

Quasi-Bayesian Dual Instrumental Variable Regression

Wang¹,

Zhou²,

Ren³

et al. 2021

Preprint

View full text Add to dashboard Cite

Recent years have witnessed an upsurge of interest in employing flexible machine learning models for instrumental variable (IV) regression, but the development of uncertainty quantification methodology is still lacking. In this work we present a scalable quasi-Bayesian procedure for IV regression, building upon the recently developed kernelized IV models. Contrary to Bayesian modeling for IV, our approach does not require additional assumptions on the data generating process, and leads to a scalable approximate inference algorithm with time cost comparable to the corresponding point estimation methods. Our algorithm can be further extended to work with neural network models. We analyze the theoretical properties of the proposed quasi-posterior, and demonstrate through empirical evaluation the competitive performance of our method. * Equal contribution Preprint. Under review.

show abstract

Section: B2 Proof Of Theorem 31mentioning

confidence: 99%

Section: Scalable Approximate Inference Via a Randomized Prior Trickmentioning

confidence: 99%

Quasi-Bayesian Dual Instrumental Variable Regression

Wang¹,

Zhou²,

Ren³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…The most successful family of UQ methods so far in deep learning has been based on the Bayesian framework [26][27][28][29][30][31][32][33][34][35][36][37][38]. Alternative methods are based, indicatively, on ensembles of NN optimization iterates or independently trained NNs [39][40][41][42][43][44][45][46][47][48][49][50], as well as on the evidential framework [51][52][53][54][55][56][57][58][59]. Although Bayesian methods and ensembles are thoroughly discussed in this paper, the interested reader is also directed to the recent review studies in [60][61][62][63][64][65][66][67][68][69][70][71][72] for more information.…”

Section: Motivation and Scope Of The Papermentioning

confidence: 99%

Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisons

Psaros¹,

Meng²,

Zou³

et al. 2022

Preprint

View full text Add to dashboard Cite

Neural networks (NNs) are currently changing the computational paradigm on how to combine data with mathematical laws in physics and engineering in a profound way, tackling challenging inverse and ill-posed problems not solvable with traditional methods. However, quantifying errors and uncertainties in NN-based inference is more complicated than in traditional methods. This is because in addition to aleatoric uncertainty associated with noisy data, there is also uncertainty due to limited data, but also due to NN hyperparameters, overparametrization, optimization and sampling errors as well as model misspecification. Although there are some recent works on uncertainty quantification (UQ) in NNs, there is no systematic investigation of suitable methods towards quantifying the total uncertainty effectively and efficiently even for function approximation, and there is even less work on solving partial differential equations and learning operator mappings between infinite-dimensional function spaces using NNs. In this work, we present a comprehensive framework that includes uncertainty modeling, new and existing solution methods, as well as evaluation metrics and post-hoc improvement approaches. To demonstrate the applicability and reliability of our framework, we present an extensive comparative study in which various methods are tested on prototype problems, including problems with mixed input-output data, and stochastic problems in high dimensions. In the Appendix, we include a comprehensive description of all the UQ methods employed, which we will make available as open-source library of all codes included in this framework.

show abstract

“…Equivalently, based on the results of (Lee et al, 2019), He et al (2020) show that f t can be expressed based on the random initialization f 0 . For t → ∞ this gives:…”

Section: Neural Tangent Kernelmentioning

confidence: 99%

An Overview of Uncertainty Quantification Methods for Infinite Neural Networks

Juengermann¹,

Laasri²,

Merkle³

2022

Preprint

View full text Add to dashboard Cite

To better understand the theoretical behavior of large neural networks, several works have analyzed the case where a network's width tends to infinity. In this regime, the effect of random initialization and the process of training a neural network can be formally expressed with analytical tools like Gaussian processes and neural tangent kernels. In this paper, we review methods for quantifying uncertainty in such infinite-width neural networks and compare their relationship to Gaussian processes in the Bayesian inference framework. We make use of several equivalence results along the way to obtain exact closed-form solutions for predictive uncertainty.

show abstract

Bayesian Deep Ensembles via the Neural Tangent Kernel

Cited by 14 publications

References 27 publications

Quasi-Bayesian Dual Instrumental Variable Regression

Quasi-Bayesian Dual Instrumental Variable Regression

Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisons

An Overview of Uncertainty Quantification Methods for Infinite Neural Networks

Contact Info

Product

Resources

About