The meaning of significant mean group differences for biomarker discovery

Abstract

Over the past decade, biomarker discovery has become a key goal in psychiatry to aid in the more reliable diagnosis and prognosis of heterogeneous psychiatric conditions and the development of tailored therapies. Nevertheless, the prevailing statistical approach is still the mean group comparison between “cases” and “controls,” which tends to ignore within-group variability. In this educational article, we used empirical data simulations to investigate how effect size, sample size, and the shape of distributions impact the interpretation of mean group differences for biomarker discovery. We then applied these statistical criteria to evaluate biomarker discovery in one area of psychiatric research—autism research. Across the most influential areas of autism research, effect size estimates ranged from small (d = 0.21, anatomical structure) to medium (d = 0.36 electrophysiology, d = 0.5, eye-tracking) to large (d = 1.1 theory of mind). We show that in normal distributions, this translates to approximately 45% to 63% of cases performing within 1 standard deviation (SD) of the typical range, i.e., they do not have a deficit/atypicality in a statistical sense. For a measure to have diagnostic utility as defined by 80% sensitivity and 80% specificity, Cohen’s d of 1.66 is required, with still 40% of cases falling within 1 SD. However, in both normal and nonnormal distributions, 1 (skewness) or 2 (platykurtic, bimodal) biologically plausible subgroups may exist despite small or even nonsignificant mean group differences. This conclusion drastically contrasts the way mean group differences are frequently reported. Over 95% of studies omitted the “on average” when summarising their findings in their abstracts (“autistic people have deficits in X”), which can be misleading as it implies that the group-level difference applies to all individuals in that group. We outline practical approaches and steps for researchers to explore mean group comparisons for the discovery of stratification biomarkers.

Management of Autism Spectrum Disorder in Italian Units of Child and Adolescent Mental Health: Diagnostic and Referral Pathways

“…However, it is important to consider that biomedical investigations as those here described are time- and cost-effective, and an accurate assessment of the predictive value of each new analysis has to be performed to preserve public health efficacy and efficiency. Similar considerations were reported in the international debate about potential ASD biomarker candidates [ 29 ]. As research progresses, genetic testing may contribute to identifying effective interventions related to specific aetiologies.…”

The link between autism and sex-specific neuroanatomy, and associated cognition and gene expression

“…We emphasize that the majority of autistic females were correctly classified and we cannot derive individual-level predictions of autism given the overlapping distributions across NT and autistic individuals. This highlights the large heterogeneity inherent to autism and the importance of identifying biologically meaningful sub-groups (57, 58). Among these, at least one sub-group is characterized by multivariate features in brain structure that are more similar to those of neurotypical males.…”

Voice patterns as markers of schizophrenia: building a cumulative generalizable approach via a cross-linguistic and meta-analysis based investigation

“…Future efforts should thus be directed to build a large open cross-linguistic corpus of speech recordings of patients with schizophrenia able to capture linguistic, cultural, socio-economic and clinical variability. This multilingual corpus may represent an ideal benchmark dataset for testing the reliability and generalizability (e.g., out of sample predictability) of voice analysis results in schizophrenia 73 , and the necessary ground for assessing its clinical applicability 74 . Not least, future studies should focus more on cross-diagnostic comparisons aimed at capturing symptom dimensions which extend over a single disorder 75,76 , and implement longitudinal designs able to test more complex hypothesis on the interaction between antipsychotic medication type and dosage, clinical (e.g., illness severity and duration) and sociodemographic (e.g., sex differences 77 ) characteristics, and speech production.…”

“…Future efforts should thus be directed to build a large open cross-linguistic corpus of speech recordings of patients with schizophrenia able to capture linguistic, cultural, socio-economic and clinical variability. This multilingual corpus may represent an ideal benchmark dataset for testing the reliability and generalizability (e.g., out of sample predictability) of voice analysis results in schizophrenia 70 , and the necessary ground for assessing its clinical applicability 71 .…”