Systematic misestimation of machine learning performance in neuroimaging studies of depression

Abstract

We currently observe a disconcerting phenomenon in machine learning studies in psychiatry: While we would expect larger samples to yield better results due to the availability of more data, larger machine learning studies consistently show much weaker performance than the numerous small-scale studies. Here, we systematically investigated this effect focusing on one of the most heavily studied questions in the field, namely the classification of patients suffering from Major Depressive Disorder (MDD) and healthy controls based on neuroimaging data. Drawing upon structural MRI data from a balanced sample of N = 1868 MDD patients and healthy controls from our recent international Predictive Analytics Competition (PAC), we first trained and tested a classification model on the full dataset which yielded an accuracy of 61%. Next, we mimicked the process by which researchers would draw samples of various sizes (N = 4 to N = 150) from the population and showed a strong risk of misestimation. Specifically, for small sample sizes (N = 20), we observe accuracies of up to 95%. For medium sample sizes (N = 100) accuracies up to 75% were found. Importantly, further investigation showed that sufficiently large test sets effectively protect against performance misestimation whereas larger datasets per se do not. While these results question the validity of a substantial part of the current literature, we outline the relatively low-cost remedy of larger test sets, which is readily available in most cases.

Population modeling with machine learning can enhance measures of mental health

“…supplement 1 ) with profiles of performance similar to the latest benchmarks on model complexity in the UKBB [ 72 ]. Moreover, simulations and empirical findings suggest that larger testing sets are more effective at mitigating optimistic performance estimates [ 53 , 73 ]. Together, this provided a pragmatic solution to the inference-prediction dilemma [ 59 , 74 ] given the 2 objectives of the present investigation to obtain reasonably good predictive models while at the same time performing parameter inference of statistical models developed on the left-out data.…”

Evaluating the performance of machine learning models for automatic diagnosis of patients with schizophrenia based on a single site dataset of 440 participants

“…In this study, we used a large single-site dataset to build SVMs to classify patients with schizophrenia and healthy controls based on brain-wide FC, with an accuracy of 85%. In contrast to recent concerns about the biased estimations of classification performance in studies with small samples [ 23 ], the present results may provide a robust estimation of SVMs for automatic diagnosis of patients with schizophrenia based on brain-wise FCs. On the basis of our data, we recommend AAL-3 for the calculation of brain-wide FC because it yielded higher classification accuracy than AAL-2 and Shen’s 268.…”

Individual prediction of trauma-focused psychotherapy response in youth with posttraumatic stress disorder using resting-state functional connectivity

“…We also investigated how the usage of a different validation approach (leave-one-out cross-validation) would influence the performance estimates of the best performing network: we observed an accuracy of 69.42%, AUC of 0.77, sensitivity of 80.95%, and specificity of 57.89%. However, given that there are theoretical and empirical reasons for why leave-one-out cross-validation is not recommended – especially in the case of small sample sizes – the reported results from this validation scheme should not be the focus of this study ( Flint et al, 2021 , Poldrack et al, 2019 ). To illustrate that there is a significant advantage in utilizing multivariate instead of univariate models we performed an experiment in which we selected the best separating voxel (according to a t -test performed on the training set) of the above network and trained and tested our linear SVM models using only this one voxel.…”