Prediction and inference diverge in biomedicine: Simulations and real-world data

Bzdok, Danilo; Engemann, Denis A.; Grisel, Olivier; Varoquaux, Gaël; Thirion, Bertrand

doi:10.1101/327437

Cited by 12 publications

(10 citation statements)

References 43 publications

(44 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This raises the question if the specificity of a Riemannian model could be enhanced in a similar way. Ultimately, what model to prefer, therefore, clearly depends on the strategic goal of the analysis (Bzdok et al, 2018;Bzdok and Ioannidis, 2019) and cannot be globally decided.…”

Section: Resultsmentioning

confidence: 99%

Predictive regression modeling with MEG/EEG: from source power to signals and cognitive states

Sabbagh

Ablin

Varoquaux

et al. 2019

Preprint

Self Cite

View full text Add to dashboard Cite

Predicting biomedical outcomes from Magnetoencephalography and Electroencephalography (M/EEG) is central to applications like decoding, brain-computer-interfaces (BCI) or biomarker development and is facilitated by supervised machine learning. Yet most of the literature is concerned with within-subject classification. Here, we focus on predicting continuous outcomes from M/EEG signal power across subjects. Considering different generative mechanisms for M/EEG signals and the biomedical outcome, we propose statistically-consistent predictive models that avoid sourcereconstruction based on the covariance as representation. Our mathematical analysis and ground truth simulations demonstrated that consistent parameter estimation can be obtained with Source Power Comodulation (SPoC) supervised spatial filtering or by embedding with Riemannian geometry. Additional simulations revealed that Riemannian methods were more robust to model violations, in particular geometric distortions induced by individual anatomy. To estimate the relative contribution of brain dynamics and anatomy to prediction performance, we propose a novel model inspection procedure based on biophysical forward modeling. Applied to cross-subject prediction, the analysis revealed that the Riemannian model better exploited anatomical information while sensitivity to brain dynamics was similar across methods. We then probed the robustness of the models across different data cleaning options. Environmental denoising was globally important but Riemannian models were strikingly robust and continued performing even without preprocessing. Our results suggest each method has its niche: SPoC is practical for within-subject prediction while the Riemannian model may enable simple end-to-end learning.

show abstract

Section: Resultsmentioning

confidence: 99%

Predictive regression modeling with MEG/EEG: from source power to signals and cognitive states

Sabbagh

Ablin

Varoquaux

et al. 2019

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Furthermore, our work stands up to potential criticism put forward by recent epistemological studies, which warn about the divergence of methods based on prediction versus inference (Lo et al, 2015;Bzdok et al, 2018), as well as the risk of misrepresentation when aggregating data across participants (Fisher et al, 2018;Smith and Little, 2018). In our study, multivariate predictive analysis converged with mass-univariate inference that was reproducible across participants ( Fig.…”

Section: Discussionmentioning

confidence: 85%

Semantic coding in the occipital cortex of early blind individuals

Abboud¹,

Engemann²,

Cohen³

2019

Preprint

Self Cite

View full text Add to dashboard Cite

The visual cortex of early blind individuals is reorganized to support cognitive functions distinct from vision. Research suggests that one such prominent function is language. However, it is unknown whether the visual cortex of blind individuals codes for word meaning. We addressed this question by comparing neuronal activity evoked by a semantic decision task, using magnetoencephalography (MEG), between 12 early blind and 14 sighted participants otherwise comparable with regard to gender, age and education. We found that average brain responses to thousands of auditory word stimuli followed similar time courses in blind and sighted participants. However, in blind participants only, we found a sustained enhancement of activity in the visual cortex. Moreover, across the whole brain, we found an effect of semantic category from about 400 ms after word onset. Strikingly, in blind participants, semantic categories were discriminable starting 580 ms after word onset from signal captured by sensors sensitive to the visual cortex. We replicated the analyses in time windows locked to stimulus onset and behavioral response, using both classical hypothesis testing and machine learning for singletrial classification. Semantic decisions were well classified in all participants (AUC ~ 0.60), but generalization capacity across participants was found reduced in the blind group due to a larger variability of discriminative patterns. In conclusion, our findings suggest that brain plasticity reorganizes the semantic system of blind individuals, and extends semantic computation into the visual cortex.

show abstract

“…Recently, Bzdok, Altman, and Krzywinski (2018) highlighted the diverging end results between classical statistics and machine learning, with the former drawing population level inferences from samples, whereas the latter is seeking to find generalisable predictive patterns. In further work, Bzdok, Engemann, Grisel, Varoquaux, and Thirion (2018) showed that low p‐values do not necessarily translate to generalisable biomarkers in out‐of‐sample data. Our model offers a robust and interpretable framework by which we can visualise how biomarkers dynamically interact and how those interactions affect the decision boundaries of classifying participants as stable MCI or progressive MCI.…”

Section: Discussionmentioning

confidence: 99%

Nonlinear biomarker interactions in conversion from mild cognitive impairment to Alzheimer's disease

Popescu

Whittington

Gunn

et al. 2020

Human Brain Mapping

View full text Add to dashboard Cite

Multiple biomarkers can capture different facets of Alzheimer's disease. However, statistical models of biomarkers to predict outcomes in Alzheimer's rarely model nonlinear interactions between these measures. Here, we used Gaussian Processes to address this, modelling nonlinear interactions to predict progression from mild cognitive impairment (MCI) to Alzheimer's over 3 years, using Alzheimer's Disease Neuroimaging Initiative (ADNI) data. Measures included: demographics, APOE4 genotype, CSF (amyloid‐β42, total tau, phosphorylated tau), [18F]florbetapir, hippocampal volume and brain‐age. We examined: (a) the independent value of each biomarker; and (b) whether modelling nonlinear interactions between biomarkers improved predictions. Each measured added complementary information when predicting conversion to Alzheimer's. A linear model classifying stable from progressive MCI explained over half the variance (R2 = 0.51, p < .001); the strongest independently contributing biomarker was hippocampal volume (R2 = 0.13). When comparing sensitivity of different models to progressive MCI (independent biomarker models, additive models, nonlinear interaction models), we observed a significant improvement (p < .001) for various two‐way interaction models. The best performing model included an interaction between amyloid‐β‐PET and P‐tau, while accounting for hippocampal volume (sensitivity = 0.77, AUC = 0.826). Closely related biomarkers contributed uniquely to predict conversion to Alzheimer's. Nonlinear biomarker interactions were also implicated, and results showed that although for some patients adding additional biomarkers may add little value (i.e., when hippocampal volume is high), for others (i.e., with low hippocampal volume) further invasive and expensive examination may be warranted. Our framework enables visualisation of these interactions, in individual patient biomarker ‘space', providing information for personalised or stratified healthcare or clinical trial design.

show abstract

Prediction and inference diverge in biomedicine: Simulations and real-world data

Cited by 12 publications

References 43 publications

Predictive regression modeling with MEG/EEG: from source power to signals and cognitive states

Predictive regression modeling with MEG/EEG: from source power to signals and cognitive states

Semantic coding in the occipital cortex of early blind individuals

Nonlinear biomarker interactions in conversion from mild cognitive impairment to Alzheimer's disease

Contact Info

Product

Resources

About