$σ$-Ridge: group regularized ridge regression via empirical Bayes noise level cross-validation

Abstract

Features in predictive models are not exchangeable, yet common supervised models treat them as such. Here we study ridge regression when the analyst can partition the features into K groups based on external side-information. For example, in high-throughput biology, features may represent gene expression, protein abundance or clinical data and so each feature group represents a distinct modality. The analyst's goal is to choose optimal regularization parameters λ = (λ1, . . . , λK ) -one for each group. In this work, we study the impact of λ on the predictive risk of group-regularized ridge regression by deriving limiting risk formulae under a high-dimensional random effects model with p n as n → ∞. Furthermore, we propose a data-driven method for choosing λ that attains the optimal asymptotic risk: The key idea is to interpret the residual noise variance σ 2 , as a regularization parameter to be chosen through cross-validation. An empirical Bayes construction maps the one-dimensional parameter σ to the K-dimensional vector of regularization parameters, i.e., σ → λ(σ). Beyond its theoretical optimality, the proposed method is practical and runs as fast as cross-validated ridge regression without feature groups (K = 1). 2 2 /2 [Hoerl and Kennard, 1970,

Co-data learning

“…One of the main assumptions underlying generalised linear models is that all variables are exchangeable. In many high-dimensional settings, however, this assumption is questionable (Ignatiadis and Lolas, 2020). For example, in cancer genomics, variables may be grouped according to some biological function.…”

“…Their results show that the uncertainty intervals of the hybrid model and full Bayes alternative correspond well, in the sense that the acquired credible intervals are competitive in terms of frequentist coverage probabilities. In light of this, more recently, Ignatiadis and Lolas (2020) proposed to tune one global parameter and to view the local parameters as function of the global parameter in a group-regularised ridge setting (see Section 1.4.2). They proved that this approach matches the best predictive performance among the class of group-regularised ridge optimisers for a family of models including the linear model.…”