Bayes and empirical-Bayes multiplicity adjustment in the variable-selection problem

Abstract

This paper studies the multiplicity-correction effect of standard Bayesian variable-selection priors in linear regression. Our first goal is to clarify when, and how, multiplicity correction happens automatically in Bayesian analysis, and to distinguish this correction from the Bayesian Ockham's-razor effect. Our second goal is to contrast empirical-Bayes and fully Bayesian approaches to variable selection through examples, theoretical results and simulations. Considerable differences between the two approaches are found. In particular, we prove a theorem that characterizes a surprising aymptotic discrepancy between fully Bayes and empirical Bayes. This discrepancy arises from a different source than the failure to account for hyperparameter uncertainty in the empirical-Bayes estimate. Indeed, even at the extreme, when the empirical-Bayes estimate converges asymptotically to the true variable-inclusion probability, the potential for a serious difference remains.to generate insights about complex, high-dimensional systems. Still, the results of such studies are often used to buttress scientific conclusions or guide policy decisions-conclusions or decisions that may be quite wrong if the implicit multiple-testing problem is ignored.Our first objective is to clarify how multiplicity correction enters Bayesian variable selection: by allowing the choice of prior model probabilities to depend upon the data in an appropriate way. Some useful references on this idea include Waller and Duncan (. We also clarify the difference between multiplicity correction and the Bayesian Ockham's-razor effect [see Jefferys and Berger (1992)], which induces a very different type of penalty on model complexity. This discussion will highlight the fact that not all Bayesian analyses automatically adjust for multiplicity.Our second objective is to describe and investigate a peculiar discrepancy between fully Bayes and empirical-Bayes variable selection. This discrepancy seems to arise from a different source than the failure to account for uncertainty in the empirical-Bayes estimate-the usual issue in such problems. Indeed, even when the empirical-Bayes estimate converges asymptotically to the true hyperparameter value, the potential for a serious difference remains.The existence of such a discrepancy between fully Bayesian answers and empirical-Bayes answers-especially one that persists even in the limitis of immediate interest to Bayesians, who often use empirical Bayes as a computational simplification. But the discrepancy is also of interest to non-Bayesians for at least two reasons.First, frequentist complete-class theorems suggest that if an empirical-Bayes analysis does not approximate some fully Bayesian analysis, then it may be suboptimal and needs alternative justification. Such justifications can be found for a variety .Second, theoretical and numerical investigations of the discrepancy revealed some unsettling properties of the standard empirical-Bayes analysis in variable selection. Of most concern is that empirical Bayes has t...

Fine Mapping Causal Variants with an Approximate Bayesian Method Using Marginal Test Statistics

“…There are different ways to set the model priors. For example, we can use a binomial prior for the number of causal SNPs, which assumes that the probability of each SNP being causal is independent and equal (Guan and Stephens 2011;Hormozdiari et al 2014 where #ðjM c jÞ is the total number of models with model size jM c j: We can also use a Beta-binomial distribution to introduce uncertainty on p; as discussed in Scott and Berger (2010). Different implications between the binomial prior and the Beta-binomial prior were also discussed and summarized in table 2 of Wilson et al (2010).…”

“…When all SNPs are included in the set, the probability r is the posterior probability that there is at least 1 causal SNP in the region. Because direct maximization of r by selecting k SNPs from p SNPs is computationally prohibitive, we follow the same stepwise selection as in Hormozdiari et al where #ðjM c jÞ is the total number of models with model size jM c j: We can also use a Beta-binomial distribution to introduce uncertainty on p; as discussed in Scott and Berger (2010). Different implications between the binomial prior and the Beta-binomial prior were also discussed and summarized in table 2 of Wilson et al (2010).…”

A sparse hierarchical Bayesian model for detecting relevant antigenic sites in virus evolution

“…This is a more general model, which subsumes a fixed π as a limiting case for α π β π /((α π + β π ) 2 (α π + β π + 1)) → 0 and has also been shown to act as a multiplicity correction in Scott and Berger (2010).…”

Is There Really a Global Business Cycle? A Dynamic Factor Model with Stochastic Factor Selection

“…(2010) show that -in contrast to a posterior density for σ 2 ε j obtained when imposing a standard Inverse Gamma prior on the variance parameter σ 2 ε j -the posterior density of σ ε j is not very sensitive to the hyperparameters of the Gaussian distribution and is not pushed away from zero when σ 2 ε j = 0. 7 The reason that we check the robustness of our results to alternative priors for p 0 is that, as noted by Scott and Berger (2010), the prior choice p 0 = 0.5 does not provide multiplicity control for the Bayesian variable -in our case, factorselection. When the number of possible variables is large and each of the binary indicators has a prior probability p 0 = 0.5 of being equal to one, the fraction of selected variables will very likely be around 0.5.…”