Bootstrap aggregating improves the generalizability of Connectome Predictive Modelling

Abstract

<jats:p>It is a long-standing goal of neuroimaging to produce reliable generalized models of brain behavior relationships. More recently data driven predicative models have become popular. Overfitting is a common problem with statistical models, which impedes model generalization. Cross validation (CV) is often used to give more balanced estimates of performance. However, CV does not provide guidance on how best to apply the models generated out-of-sample. As a solution, this study proposes an ensemble learning method, in this case bootstrap aggregating, or bagging, encompassing both model parameter estimation and feature selection. Here we investigate the use of bagging when generating predictive models of fluid intelligence (fIQ) using functional connectivity (FC). We take advantage of two large openly available datasets, the Human Connectome Project (HCP), and the Philadelphia Neurodevelopmental Cohort (PNC). We generate bagged and non-bagged models of fIQ in the HCP. Over various test-train splits, these models are evaluated in sample, on left out HCP data, and out-of-sample, on PNC data. We find that in sample, a non-bagged model performs best, however out-of-sample the bagged models perform best. We also find that feature selection can vary substantially within-sample. A more considered approach to feature selection, alongside data driven predictive modeling, is needed to improve cross sample performance of FC based brain behavior models.</jats:p>

Connectome‐based predictive modelling of cognitive reserve using task‐based functional connectivity

“…As the test set resting-state scan contained only 200 time points, more time points may be needed for connectivity matrices to have sufficient variation across individuals in order to accurately predict complex phenotypes such as CR. Advanced modelling techniques, such as bootstrap aggregating (O'Connor et al, 2020) and partial least squares regression (Yoo et al, 2018), when implemented within CPM frameworks have also been shown to improve generalizability to external datasets.…”

Brain connectivity at rest predicts individual differences in normative activity during movie watching

“…The linear models from all 100 iterations were then combined into a single model following a bootstrap aggregating (“bagging”) approach that has been shown to reduce overfitting and improve model accuracy (Fig. 1B, step 5; O’Connor et al, 2020). This bagged model was then used to predict Day 2 gISC from Day 2 RSFC (Fig.…”

“…Because the HCP 7T dataset is composed of data from individuals of varying degrees of genetic relatedness (monozygotic and dizygotic twins, non-twin siblings, and unrelated individuals; 93 unique families), all individuals from the same family were randomly assigned to one of two groups of 88 (i.e., split-half cross-validation), with one group being used to train a model that would then be tested on the other (and vice versa). The following approach was then applied to 100 of these random splits of the data to assess the performance of rCPM across different training/testing sets and to build a bagged model that is more robust to overfitting (O’Connor et al, 2020). Calculate leave-one-out (LOO) ISC according to the method described in section 2.4 separately for each group of 88 subjects.…”

“…It could then be the case that the CV models rely on stimulus-specific signals and may fail to predict gISC during viewing of a different movie. A bootstrap aggregating, or “bagging,” approach was used to test whether the 200 linear models trained on Day 1 movie watching and resting state data could predict Day 2 gISC (derived from a different set of stimuli) from Day 2 RSFC, as previous work has shown bagged CPM models to be more accurate and more generalizable than their non-bagged counterparts (O’Connor et al, 2020). To construct the bagged model, RSFC edges that passed the P < .01 feature selection step in at least 10% (20/200, reflecting the 100 iterations of split-half cross-validation) of iterations were identified, yielding 1,618 edges total.…”

Connectome-based predictive modeling of cognitive reserve using task-based functional connectivity

“…As the test set resting-state scan contained only 200 time points, more time points may be needed for connectivity matrices to have sufficient variation across individuals in order to accurately predict complex phenotypes such as CR. Advanced modeling techniques, such as bootstrap aggregating (O’Connor et al, 2020) and partial least squares regression (Yoo et al, 2018), when implemented within CPM frameworks have also been shown to improve generalizability to external datasets.…”