Eliminating accidental deviations to minimize generalization error and maximize replicability: Applications in connectomics and genomics

Abstract

Replicability, the ability to replicate scientific findings, is a prerequisite for scientific discovery and clinical utility. Troublingly, we are in the midst of a replicability crisis. A key to replicability is that multiple measurements of the same item (e.g., experimental sample or clinical participant) under fixed experimental constraints are relatively similar to one another. Thus, statistics that quantify the relative contributions of accidental deviations—such as measurement error—as compared to systematic deviations—such as individual differences—are critical. We demonstrate that existing replicability statistics, such as intra-class correlation coefficient and fingerprinting, fail to adequately differentiate between accidental and systematic deviations in very simple settings. We therefore propose a novel statistic, discriminability, which quantifies the degree to which an individual’s samples are relatively similar to one another, without restricting the data to be univariate, Gaussian, or even Euclidean. Using this statistic, we introduce the possibility of optimizing experimental design via increasing discriminability and prove that optimizing discriminability improves performance bounds in subsequent inference tasks. In extensive simulated and real datasets (focusing on brain imaging and demonstrating on genomics), only optimizing data discriminability improves performance on all subsequent inference tasks for each dataset. We therefore suggest that designing experiments and analyses to optimize discriminability may be a crucial step in solving the replicability crisis, and more generally, mitigating accidental measurement error.

State-dependent representations of mixtures by the olfactory bulb

“…The ability to reliably encode the presence of EB was first quantified based on within-class vs. across-class correlation by counting the proportion of trials where within-class correlation was higher than across-class ( Bridgeford et al, 2021 ). This indicated that M/T cells from anaesthetised mice showed better discriminability initially, but later, M/T cells in the behaving mice performed as well as the anaesthetised mice ( Figure 5C ).…”

State-dependent representations of mixtures by the olfactory bulb

“…The ability to reliably encode the presence of ethyl butyrate was first quantified based on within-class vs. across-class correlation, by counting the proportion of trials where within-class correlation was higher than across-class (Bridgeford et al, 2021). This indicated that M/T cells from anaesthetised mice showed better discriminability initially, but later, M/T cells in the behaving mice performed as well as the anaesthetised mice (Fig.…”

“…Discriminability index based on correlation coefficient was calculated in the same way as Dicr , described in (Bridgeford et al, 2021), except that the distance measure used was 1 – Pearson’s correlation coefficient, instead of the Euclidian distance.…”

“…Deviation from linearity was the difference between an observed mixture response amplitude and a linear sum of components normalized by the joint standard deviation: (Robserved -Rlinear sum)/(s.e.m.observed + s.e.m.linearsum) Discriminability index based on correlation coefficient was calculated in the same way as Dicr, described in (Bridgeford et al, 2021), except that the distance measure used was 1 -Pearson's correlation coefficient, instead of the Euclidian distance.…”

Neuroimaging brain growth charts: A road to mental health

“…A recently proposed method, named discriminability, defined as the fraction of frequency at which the similarity across-subject measurements is smaller than similarity within-subject measurements, can assess the reliability of multivariate data more flexibly and be used to any stage of data processing (Wang et al ., 2020 ). This method is based on nonparametric energy statistics (Rizzo and Székely, 2016 ) and kernel mean embeddings (Muandet et al ., 2016 ) approaches, and it is equivalent to ICC under the Gaussian assumption for univariate data (Wang et al ., 2020 ; Bridgeford et al ., 2021 ; Milham et al ., 2021 ). Methods of data collection and analysis can contribute to low reliability in neuroimaging studies.…”