Bayesian Models for Sparse Regression Analysis of High Dimensional Data*

Abstract

This paper considers the task of building efficient regression models for sparse multivariate analysis of high dimensional data sets, in particular it focuses on cases where the numbers q of responses Y = (y k , 1 ≤ k ≤ q) and p of predictors X = (x j , 1 ≤ j ≤ p) to analyse jointly are both large with respect to the sample size n, a challenging bi-directional task. The analysis of such data sets arise commonly in genetical genomics, with X linked to the DNA characteristics and Y corresponding to measurements of fundamental biological processes such as transcription, protein or metabolite production. Building on the Bayesian variable selection setup for the linear model and associated efficient MCMC algorithms developed for single responses, we discuss the generic framework of hierarchical related sparse regressions, where parallel regressions of y k on the set of covariates X are linked in a hierarchical fashion, in particular through the prior model of the variable selection indicators γ kj , which indicate among the covariates x j those which are associated to the response y k in each multivariate regression. Structures for the joint model of the γ kj , which correspond to different compromises between the aims of controlling sparsity and that of enhancing the detection of predictors that are associated with many responses ('hot spots'), will be discussed and a new multiplicative model for the probability structure of the γ kj will be presented. To perform inference for these models in high dimensional setups , novel adaptive MCMC algorithms are needed. As sparsity is paramount and most of the associations expected to be zero, new algorithms that progressively focus on part of the space where the most interesting associations occur are of great interest. We shall discuss their formulation and theoretical properties, and demonstrate their use on simulated and real data from genomics.

Bayesian Detection of Expression Quantitative Trait Loci Hot Spots

“…In the following we provide some technical details omitted from the main text of the local and global moves that we found useful to implement. At each sweep of the algorithm each/both of moves can be applied to all the q regression equations or to a random without replacement subgroup of them (see Richardson et al (2011) for alternative subgroup selection with adaptive probability).…”

Some of Statistics Canada’s Contributions to Survey Methodology

“…Adaptive MCMC algorithms have recently been used in a number of different statistical applications, and often lead to significant speed-ups, even in hundreds of dimensions; see, e.g., Rosenthal (2009), Craiu et al (2009), Giordani and Kohn (2010) and Richardson et al (2011). On the other hand, adaptive MCMC algorithms use previous iterations to determine their future transitions, so they violate the Markov property which provides the justification for conventional MCMC.…”

“…In particular, Roberts and Rosenthal (2007) show that adaptive algorithms will still converge to the target density π provided they satisfy two fairly mild conditions: "Diminishing Adaptation" (the algorithm adapts by less and less as time goes on), and "Containment" (the chain never gets too lost, in the sense that it remains bounded in probability). Conditions such as these have been used to formally justify adaptive algorithms in many examples; see, e.g., Roberts and Rosenthal (2009) and Richardson et al (2011). Adaptive MCMC appears to hold great promise for improving statistical computation in many application areas in the years ahead.…”

Geometric ergodicity of random scan Gibbs samplers for hierarchical one-way random effects models

“…Gibbs samplers can be implemented in either a deterministic scan or a random scan, among other variants (Johnson et al, 2013;Liu et al, 1994). Although deterministic scan MCMC algorithms are currently popular in the statistics literature, random scan algorithms were some of the first used in MCMC settings (Geman and Geman, 1984;Metropolis et al, 1953) and remain useful in applications (Lee et al, 2013;Richardson et al, 2010). Random scan Gibbs samplers can also be implemented adaptively while the deterministic scan version cannot.…”