Comparing and Weighting Imperfect Models Using D-Probabilities

Li, Meng; Dunson, David B.

doi:10.1080/01621459.2019.1611140

Cited by 12 publications

(8 citation statements)

References 51 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Probabilistic averaging of models (PAM) is a model averaging strategy that is able to capture this predictive uncertainty. PAM was introduced by Akaike in 1978 and is also known as 'model averaging' [31,32]. However, since different notions of model averaging have been established in the literature [17,18,[33][34][35], we refer to the averaging of predictive distributions as PAM.…”

Section: Probabilistic Averaging Of Modelsmentioning

confidence: 99%

Treatment response prediction: Is model selection unreliable?

Augustin

Wang

Walz

et al. 2022

Preprint

View full text Add to dashboard Cite

Quantitative modelling has become an essential part of the drug development pipeline. In particular, pharmacokinetic and pharmacodynamic models are used to predict treatment responses in order to optimise clinical trials and assess the safety and efficacy of dosing regimens across patients. It is therefore crucial that treatment response predictions are reliable. However, the data available to fit models are often limited, which can leave considerable uncertainty about the best model to use. Common practice is to select the model that is most consistent with the observed data based on the Akaike information criterion (AIC). Another popular approach is to average the predictions across the subset of models consistent with the data. In this article, we argue that both approaches can lead to unreliable predictions, as treatment responses typically display nonlinear dynamics, so models can be consistent with the observed dynamics, whilst predicting incorrect treatment responses. This is especially the case when predicting treatment responses for either times or dosing regimens that go beyond the observed dynamics. Across a range of experiments on both real laboratory data and synthetically derived data onNeisseria gonorrhoeaeresponse to ciprofloxacin, we show that probabilistic averaging of models results in more reliable treatment response predictions.

show abstract

Section: Probabilistic Averaging Of Modelsmentioning

confidence: 99%

Treatment response prediction: Is model selection unreliable?

Augustin

Wang

Walz

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…van der Vaart and van Zanten (2009) proposed a fully Bayesian scheme by endowing the standard error σ with a hyperprior, which is supported on a compact interval [a, b] ⊂ (0, ∞) that contains σ 0 with a Lebesgue density bounded away from zero. This approach has been followed by many others (Bhattacharya et al, 2014;Li and Dunson, 2020). de Jonge and van Zanten (2013) showed a Bernstein-von Mises theorem for the marginal posterior of σ, where the prior for σ is relaxed to be supported on (0, ∞).…”

Section: Unknown Error Variancementioning

confidence: 99%

Equivalence of Convergence Rates of Posterior Distributions and Bayes Estimators for Functions and Nonparametric Functionals

Liu,

2020

Preprint

Self Cite

View full text Add to dashboard Cite

We study the posterior contraction rates of a Bayesian method with Gaussian process priors in nonparametric regression and its plug-in property for differential operators. For a general class of kernels, we establish convergence rates of the posterior measure of the regression function and its derivatives, which are both minimax optimal up to a logarithmic factor for functions in certain classes. Our calculation shows that the rate-optimal estimation of the regression function and its derivatives share the same choice of hyperparameter, indicating that the Bayes procedure remarkably adapts to the order of derivatives and enjoys a generalized plug-in property that extends real-valued functionals to function-valued functionals. This leads to a practically simple method for estimating the regression function and its derivatives, whose finite sample performance is assessed using simulations.Our proof shows that, under certain conditions, to any convergence rate of Bayes estimators there corresponds the same convergence rate of the posterior distributions (i.e., posterior contraction rate), and vice versa. This equivalence holds for a general class of Gaussian processes and covers the regression function and its derivative functionals, under both the L 2 and L ∞ norms. In addition to connecting these two fundamental large sample properties in Bayesian and non-Bayesian regimes, such equivalence enables a new routine to establish posterior contraction rates by calculating convergence rates of nonparametric point estimators.At the core of our argument is an operator-theoretic framework for kernel ridge regression and equivalent kernel techniques. We derive a range of sharp non-asymptotic bounds that are pivotal in establishing convergence rates of nonparametric point estimators and the equivalence theory, which may be of independent interest.

show abstract

“…Moreover, George Box's famous adage "all models are wrong" may then be invoked to question the use of these "M-closed tools" in any practical application. For instance, Li and Dunson (2016) argue that "Philosophically, in order to interpret pr(M j | y (n) ) as a model probability, one must rely on the (arguably always flawed) assumption that one of the models in the list M is exactly true, known as the M-closed case. "…”

Section: Loo Is Motivated By An Illusory Distinction Between M-open Tmentioning

confidence: 99%

Rejoinder: More Limitations of Bayesian Leave-One-Out Cross-Validation

Gronau¹,

Wagenmakers²

2018

Preprint

View full text Add to dashboard Cite

We recently discussed several limitations of Bayesian leave-one-out cross-validation (LOO) for model selection. Our contribution attracted three thought-provoking commentaries. In this rejoinder, we address each of the commentaries and identify several additional limitations of LOO-based methods such as Bayesian stacking. We focus on differences between LOO-based methods versus approaches that consistently use Bayes' rule for both parameter estimation and model comparison. We conclude that LOO-based methods do not align satisfactorily with the epistemic goal of mathematical psychology.

show abstract

Comparing and Weighting Imperfect Models Using D-Probabilities

Cited by 12 publications

References 51 publications

Treatment response prediction: Is model selection unreliable?

Treatment response prediction: Is model selection unreliable?

Equivalence of Convergence Rates of Posterior Distributions and Bayes Estimators for Functions and Nonparametric Functionals

Rejoinder: More Limitations of Bayesian Leave-One-Out Cross-Validation

Contact Info

Product

Resources

About