Better prediction by use of co‐data: adaptive group‐regularized ridge regression

Abstract

For many high-dimensional studies, additional information on the variables, like (genomic) annotation or external p-values, is available. In the context of binary and continuous prediction, we develop a method for adaptive group-regularized (logistic) ridge regression, which makes structural use of such 'co-data'. Here, 'groups' refer to a partition of the variables according to the co-data. We derive empirical Bayes estimates of group-specific penalties, which possess several nice properties: i) they are analytical; ii) they adapt to the informativeness of the co-data for the data at hand; iii) only one global penalty parameter requires tuning by cross-validation. In addition, the method allows use of multiple types of co-data at little extra computational effort.We show that the group-specific penalties may lead to a larger distinction between 'near-zero' and relatively large regression parameters, which facilitates post-hoc variable selection. The method, termed GRridge, is implemented in an easy-to-use R-package. It is demonstrated on two cancer genomics studies, which both concern the discrimination of precancerous cervical lesions from normal cervix tissues using methylation microarray data. For both examples, GRridge clearly improves the predictive performances of ordinary logistic ridge regression and the group lasso. In addition, we show that for the second study the relatively good predictive performance is maintained when selecting only 42 variables.

IPF-LASSO: IntegrativeL1-Penalized Regression with Penalty Factors for Prediction Based on Multi-Omics Data

“…In addition, having different penalty factors in IPF-LASSO allows for the incorporation of prior biological knowledge or practical concerns. To address the computational cost induced by the choice of the penalty factors for large M , alternatives to our grid search cross-validation approach may be considered in the future, for example, based on empirical Bayes procedures [20], on model selection criteria such as the Akaike information criterion (AIC) or Bayesian information criterion (BIC), or using the approach inspired from adaptive LASSO [14] adopted by Ternès et al [27] in the specific case of treatment-biomarker interactions.…”

“…In their study, published in a genetics journal and focusing on the agricultural application, they apply this method to their dataset and do not investigate it from a methodological point of view. A similar approach based on L 2 -penalized logistic regression [20] formalizes and extends this idea with the purpose to better integrate external data such as annotation or external p values.…”

Learning from a lot: Empirical Bayes for high‐dimensional model‐based prediction

“…This leads to a system of G linear equations with G unknowns. For several cancer genomics applications, van de Wiel et al (2016) and Novianti, Snoek, Wilting, and van de Wiel (2017) show that using group penalty parameters that are inverse proportional to solution̂g improves predictive performance. Note that the comparison between likelihood-based (Section 2.2) and moment-based estimation is on a somewhat different footing here than for ordinary parameter estimation.…”

“…Sometimes, it may be worthwhile to combine EB with CV. For example, if one wishes to apply different penalties g for groups of variables (Boulesteix, De Bin, Jiang, & Fuchs, 2017;van de Wiel et al, 2016), one may reparameterize g = ′ g and optimize the global parameter by CV with respect to predictive performance while estimating the multipliers ′ g by EB. Alternatively, CV or similar out-of-bag approaches may be used to tune the initial EB estimates to improve predictive performance or to implement parameter thresholding.…”

“…In the penalized setting, estimators of 2 more advanced than (14) are available (Cule & De Iorio, 2013). We proposed an alternative that accounts for the bias of̂because of penalization (van de Wiel et al, 2016)…”