Detect and Correct Bias in Multi-Site Neuroimaging Datasets

Abstract

The desire to train complex machine learning algorithms and to increase the statistical power in association studies drives neuroimaging research to use ever-larger datasets. The most obvious way to increase sample size is by pooling scans from independent studies. However, simple pooling is often ill-advised as selection, measurement, and confounding biases may creep in and yield spurious correlations. In this work, we combine 35,320 magnetic resonance images of the brain from 17 studies to examine bias in neuroimaging. In the first experiment, Name That Dataset, we provide empirical evidence for the presence of bias by showing that scans can be correctly assigned to their respective dataset with 71.5% accuracy. Given such evidence, we take a closer look at confounding bias, which is often viewed as the main shortcoming in observational studies. In practice, we neither know all potential confounders nor do we have data on them. Hence, we model confounders as unknown, latent variables. Kolmogorov complexity is then used to decide whether the confounded or the causal model provides the simplest factorization of the graphical model. Finally, we present methods for dataset harmonization and study their ability to remove bias in imaging features. In particular, we propose an extension of the recently introduced ComBat algorithm to control for global variation across image features, inspired by adjusting for population stratification in genetics. Our results demonstrate that harmonization can reduce dataset-specific information in image features. Further, confounding bias can be reduced and even turned into a causal relationship. However, harmonziation also requires caution as it can easily remove relevant subject-specific information.

Metadata Normalization

“…To perform classification on such multi-site data, the source of the data (dataset label) becomes the metadata, which is parameterized here by onehot encodings. Medical imaging datasets acquired in multiple sites with different scanning protocols is a core challenge for machine learning algorithms in medicine, [38,56], as different scanning protocols lead to different image formations. Differing class formations across sites (as in this experiment) creates a simple undesirable cue for the model to leverage during prediction as a confounder.…”

Autism excitation-inhibition imbalance linked to brain hyperconnectivity: An analysis based on 657 autistic subjects

“…We performed our subtyping analyses on 657 autistic patients from the Autism Brain Imaging Data Exchange (ABIDE) repository 48 , all of them having passed a very strict quality-assurance criterion of elimination of subjects by head movement during image acquisition, thus correcting a well-known spurious excess of functional connectivity driven by head movements, which is even more pronounced in the autistic condition. Moreover, to overcome inter-scanner variability in the functional connectivity values across different Institutions, we applied rigorous harmonization strategies to transform data that are heterogeneous --and that come from different Institutions--into equivalents [49][50][51][52] .…”