Population heterogeneity in clinical cohorts affects the predictive accuracy of brain imaging

Abstract

Brain imaging research enjoys increasing adoption of supervised machine learning for single-participant disease classification. Yet, the success of these algorithms likely depends on population diversity, including demographic differences and other factors that may be outside of primary scientific interest. Here, we capitalize on propensity scores as a composite confound index to quantify diversity due to major sources of population variation. We delineate the impact of population heterogeneity on the predictive accuracy and pattern stability in 2 separate clinical cohorts: the Autism Brain Imaging Data Exchange (ABIDE, n = 297) and the Healthy Brain Network (HBN, n = 551). Across various analysis scenarios, our results uncover the extent to which cross-validated prediction performances are interlocked with diversity. The instability of extracted brain patterns attributable to diversity is located preferentially in regions part of the default mode network. Collectively, our findings highlight the limitations of prevailing deconfounding practices in mitigating the full consequences of population diversity.

Using the Excitation/Inhibition Ratio to Optimize the Classification of Autism and Schizophrenia

“…A limitation inherent to the top-down approach we took in this study is that diagnosis overlap between ASD and SZ cannot be fully characterized given the a priori reliance on clinically pre-defined group labels (Moreau et a., 2021), especially due to the heterogeneity of ASD and SZ (Port, Oberman & Roberts, 2019; Benkarim et al, 2022). For example, it has been shown that the relationship between the E/I balance and the cerebro-cerebellar functional connectivity in ASD is not uniform across samples (Hegarty et al, 2018), therefore a bottom-up approach in a larger sample would be extremely valuable to inform on different E/I-based clinical subtypes.…”

“…While ASD is primarily characterized by impairments in social communication skills and by repetitive behaviors, a SZ diagnosis consists of positive (e.g., hallucinations, delusions) and negative (e.g., social withdrawal) symptoms. The heterogeneity of both diagnostic categories (Benkarim et al, 2022; Segal et al, 2022) and their phenotypic overlap (Kästner et al, 2015) can hinder accurate diagnosis. More precisely, ASD and SZ co-occur in approximately 4% of cases (Lai et al, 2019), and share both social (Oliver et al, 2020) and sensory-motor deficits (Du et al, 2021).…”

Relative Brain Age Is Associated with Socioeconomic Status and Anxiety/Depression Problems in Youth

“…Because we used this previously validated training set to study this question, our sample size was limited. Recent findings show that cross-validated participant level predictions in neuroimaging research vary with sociodemographic diversity in the training sample (Benkarim et al, 2022). Consistent with these findings we show that parent occupation and public assistance predicted RBA, further suggesting that socioeconomic diversity should be considered in future implementations of brain age prediction algorithms.…”

Expectations for Artificial Intelligence (AI) in Psychiatry

“…The diagnostic performance of a ML algorithm to detect tuberculosis that was developed using a chest X-ray training dataset of one population fell when used with another population [ 106 ]. Population diversity in age, sex, and brain scanning site substantially affected the predictive accuracy of ML neuroimaging studies, including for autism spectrum disorder [ 107 ]. An algorithm to predict clinical orders by hospital admission diagnosis performed better when trained on a small amount of recent data (one month) than when trained on larger amounts of older data (12 months of 3-year-old data) due to changing practice patterns [ 108 ].…”