Performance reserves in brain-imaging-based phenotype prediction

Abstract

Machine learning studies have shown that various phenotypes can be predicted from structural and functional brain images. However, in most such studies, prediction performance ranged from moderate to disappointing. It is unclear whether prediction performance will substantially improve with larger sample sizes or whether insufficient predictive information in brain images impedes further progress. Here, we systematically assess the effect of sample size on prediction performance using sample sizes far beyond what is possible in common neuroimaging studies. We project 3-9 fold improvements in prediction performance for behavioral and mental health phenotypes when moving from one thousand to one million samples. Moreover, we find that moving from single imaging modalities to multimodal input data can lead to further improvements in prediction performance, often on par with doubling the sample size. Our analyses reveal considerable performance reserves for neuroimaging-based phenotype prediction. Machine learning models may benefit much more from extremely large neuroimaging datasets than currently believed.

Socioeconomic Resources are Associated with Distributed Alterations of the Brain’s Intrinsic Functional Architecture in Youth

“…Because correlations observed in testing samples are typically lower when models are trained on a smaller training subsample (32), we additionally performed resampling tests (see Supplement) to assess whether observed performance in the low motion subsample differs significantly from performance in randomly drawn subsamples of the same size. These resampling tests were non-significant (see Table S3), indicating that the observed out-of-sample parental education correlations in the actual low motion subsample does not statistically differ from other subsamples of the same size.…”

“…Previous examinations of the associations between SER and functional connectivity patterns in the brain have largely relied on regionalist and apriorist methods (e.g., 31, 32), see (17) for a review. That is, SER-brain associations have been examined within individual pre-selected regions (e.g., amygdala – ventromedial prefrontal connectivity) based on prior theory.…”

Assessing Pain Research: A Narrative Review of Emerging Pain Methods, Their Technosocial Implications, and Opportunities for Multidisciplinary Approaches

“…While diversifying data sources may help represent pain more realistically or wholistically, attempts to scale subjectivity and ambiguity or ignore them (i.e., pain prediction models) runs a risk of disadvantaging data outliers ( 376 ) and delegitimizing the pain of those who do not fit the model. Researchers, in seeking to use or build large data sets, should also be wary of “scale thinking,” as data amount or data diversity is not necessarily proportional to model performance ( 377 ), and can be unintentionally extractive ( 337 ).…”

Comparing interpretation methods in mental state decoding analyses with deep learning models

“…For these reasons, we echo a recent call of machine learning researchers to always consider whether the application of complex models (such as DL models) is necessary to answer the research question at hand, or whether the application of simpler models, with better interpretability, could suffice (Rudin, 2019) (especially in high-stakes decision situations, such as the development of biomarker disease models). While we do believe that DL models hold a high promise for the field of mental state decoding, e.g., in their ability to learn from large-scale neuroimaging datasets (Schulz et al, 2022), we also believe that many common mental state decoding analyses, which focus on few mental states in tens to a hundred of individuals, can be well-addressed with simpler decoding models with better interpretability (e.g., Grosenick et al, 2013, Hoyos-Idrobo et al, 2018, Kriegeskorte et al, 2006, Michel et al, 2011, Schulz et al, 2020.…”