Mind the gap: performance metric evaluation in brain-age prediction

Abstract

Estimating age based on neuroimaging-derived data has become a popular approach to developing markers for brain integrity and health. While a variety of machine-learning algorithms can provide accurate predictions of age based on brain characteristics, there is significant variation in model accuracy reported across studies. We predicted age based on neuroimaging data in two population-based datasets, and assessed the effects of age range, sample size, and age-bias correction on the model performance metrics r, R2, Root Mean Squared Error (RMSE), and Mean Absolute Error (MAE). The results showed that these metrics vary considerably depending on cohort age range; r and R2 values are lower when measured in samples with a narrower age range. RMSE and MAE are also lower in samples with a narrower age range due to smaller errors/brain age delta values when predictions are closer to the mean age of the group. Across subsets with different age ranges, performance metrics improve with increasing sample size. Performance metrics further vary depending on prediction variance as well as mean age difference between training and test sets, and age-bias corrected metrics indicate high accuracy - also for models showing poor initial performance. In conclusion, performance metrics used for evaluating age prediction models depend on cohort and study-specific data characteristics, and cannot be directly compared across different studies. Since age-bias corrected metrics in general indicate high accuracy, even for poorly performing models, inspection of uncorrected model results provides important information about underlying model attributes such as prediction variance.

Brain-wide associations between white matter and age highlight the role of fornix microstructure in brain ageing

“…Simultaneously, differences between diffusion approaches, and both variance explained and prediction error (RMSE, MAE) were smaller in this study. These differences are likely due to the narrower age range in our study 45 , whereas our 11 significantly larger sample emphasises the reliability of our findings.…”

Multimodal biological brain age prediction using magnetic resonance imaging and angiography with the identification of predictive regions

“…Thus, the fact that the TOF MRA datasets include less information than the T1‐weighted MRI datasets might partly explain the difference in accuracy between the CNN T1 and CNN TOF models. When combining both modalities, the resulting mean absolute error significantly improves and is comparable to the results described in literature, with reported MAEs mostly varying between 3 to 5 years when using T1‐weighted MRI datasets (Bashyam et al, 2020; Cole et al, 2017; Jonsson et al, 2019; Levakov et al, 2020; Mouches, Wilms, Rajashekar, Langner, & Forkert, 2021; Peng et al, 2021; Wilms et al, 2020), but often using considerably more training data, and participants with a much narrower age range, which hinders a direct comparison of the results (de Lange et al, 2021). Moreover, Bashyam et al (2020) who previously trained a deep learning brain age prediction model using more than 11,000 datasets and tested it on the SHIP database reported a MAE of 4.12 years, showing that this database is rather challenging for the brain age prediction task.…”

“…First, while using a single database increases the consistency of the results and reduces biases, leading to more robust explanations, it also results in a model that is less robust to varying scanning parameters, and limits the amount of data available. Therefore, the model prediction accuracy would benefit from training using a larger sample size, as previously demonstrated in the context of brain age prediction (de Lange et al, 2021), and data collected from different centers, especially when using deep learning models, which are known to be data hungry (Marcus, 2018). Nevertheless, based on the excellent results of the SFCN architecture on the highly diverse PAC2019 brain age prediction data reported in Peng et al (2021), we assume that the general findings of this study will hold true even for multicenter datasets, especially when proper harmonization strategies are implemented to remove possible confounding biases.…”

Brain-age prediction: a systematic comparison of machine learning workflows

“…One also needs to choose from a large pool of ML algorithms, such as random forest regression (RFR), relevance vector regression (RVR), and Gaussian process regression (GPR), many of which have shown success in brain-age estimation. These choices are known to affect performance (Gutierrez Becker et al ., 2018, Baecker et al ., 2021 a ; de Lange et al ., 2022). However, previous studies have performed only limited comparisons on the same data and setup.…”

“…This age bias complicates or may even mislead downstream individualized decision-making. It can be mitigated using bias correction models; usually, linear regression predicting brain-age or delta using chronological age (Le et al ., 2018; Liang et al ., 2019, Smith et al ., 2019 b ; de Lange et al ., 2022). These correction models are also used to counter the systematic under- or over-estimation of age in a novel site, usually reflected in the non-zero average delta in healthy controls.…”