Numerical Uncertainty in Analytical Pipelines Lead to Impactful Variability in Brain Networks

Kiar, Gregory; Chatelain, Yohan; Castro, Pablo de Oliveira; Petit, Éric; Rokem, Ariel; Varoquaux, Gaël; Mišić, Bratislav; Evans, AC; Glatard, Tristan

doi:10.1101/2020.10.15.341495

“…Previous work has called into question the the reliability of neuroimaging analysis (e.g., (24,41,42)). We assessed the reliability of a specific approach, tractometry, which is grounded in decades of anatomical knowledge, and we demonstrate that this approach is reproducible, reliable and robust.…”

Section: Discussionmentioning

confidence: 99%

Evaluating the reliability of human brain white matter tractometry

Kruper

¹

,

Yeatman

²

,

Richie-Halford

³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

The validity of research results depends on the reliability of analysis methods. In recent years, there have been concerns about the validity of research that uses diffusion-weighted MRI (dMRI) to understand human brain white matter connections in vivo, in part based on reliability of the analysis methods used in this field. We defined and assessed three dimensions of reliability in dMRI-based tractometry, an analysis technique that assesses the physical properties of white matter pathways: (1) reproducibility, (2) test-retest reliability and (3) robustness. To facilitate reproducibility, we provide software that automates tractometry (https://yeatmanlab.github.io/pyAFQ). In measurements from the Human Connectome Project, as well as clinical-grade measurements, we find that tractometry has high test-retest reliability that is comparable to most standardized clinical assessment tools. We find that tractometry is also robust: showing high reliability with different choices of analysis algorithms. Taken together, our results suggest that tractometry is a reliable approach to analysis of white matter connections. The overall approach taken here both demonstrates the specific trustworthiness of tractometry analysis and outlines what researchers can do to demonstrate the reliability of computational analysis pipelines in neuroimaging.

show abstract

“…An existing dataset containing Monte Carlo Arithmetic (MCA) perturbed structural human brain networks was used for these experiments [25]. While further information on the processing and curation of this dataset can be found in [14], a brief description of the data and processing follows here.…”

Section: Datasetmentioning

confidence: 99%

“…These models often build representations upon processed imaging data, in which 3D or 4D images have been transformed into estimates of structure [8], function [9], or connectivity [10]. However, there is a lack of reliability in these estimates, including variation across analysis team [11], software library [12], operating system [13], and instability in the face of numerical noise [14]. This uncertainty limits the ability of models to learn generalizable relationships among data, and leads to biased predictors.…”

Section: Introductionmentioning

confidence: 99%

“…Importantly, this technique produces a range of equally plausible results, where no single observation is more or less valid than the others -including those which were left unperturbed. While sampling from a set of perturbed connectomes may have an impact on learning brain-phenotype relationships [14], there remains potential for leveraging the distribution of perturbed results to augment datasets in lieu of increasing sample sizes, performing repeated measurements, or developing data-specific augmentation models.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Data Augmentation Through Monte Carlo Arithmetic Leads to More Generalizable Classification in Connectomics

Kiar

¹

,

Chatelain²,

Salari³

et al. 2021

Neurons, Behavior, Data Analysis, and Theory

Self Cite

View full text Add to dashboard Cite

Machine learning models are commonly applied to human brain imaging datasets in an effort to associate function or structure with behaviour, health, or other individual phenotypes. Such models often rely on low-dimensional maps generated by complex processing pipelines. However, the numerical instabilities inherent to pipelines limit the fidelity of these maps and introduce computational bias. Monte Carlo Arithmetic, a technique for introducing controlled amounts of numerical noise, was used to perturb a structural connectome estimation pipeline, ultimately producing a range of plausible networks for each sample. The variability in the perturbed networks was captured in an augmented dataset, which was then used for an age classification task. We found that resampling brain networks across a series of such numerically perturbed outcomes led to improved performance in all tested classifiers, preprocessing strategies, and dimensionality reduction techniques. Importantly, we find that this benefit does not hinge on a large number of perturbations, suggesting that even minimally perturbing a dataset adds meaningful variance which can be captured in the subsequently designed models.

show abstract

“…These models often build representations upon processed imaging data, in which 3D or 4D images have been transformed into estimates of structure [8], function [9], or connectivity [10]. However, there is a lack of reliability in these estimates, including variation across analysis team [11], software library [12], operating system [13], and instability in the face of numerical noise [14]. This uncertainty limits the ability of models to learn generalizable relationships among data, and leads to biased predictors.…”

Section: Introductionmentioning

confidence: 99%

Data Augmentation Through Monte Carlo Arithmetic Leads to More Generalizable Classification in Connectomics

Kiar

¹

,

Chatelain

²

,

Salari

³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Machine learning models are commonly applied to human brain imaging datasets in an effort to associate function or structure with behaviour, health, or other individual phenotypes. Such models often rely on low-dimensional maps generated by complex processing pipelines. However, the numerical instabilities inherent to pipelines limit the fidelity of these maps and introduce computational bias. Monte Carlo Arithmetic, a technique for introducing controlled amounts of numerical noise, was used to perturb a structural connectome estimation pipeline, ultimately producing a range of plausible networks for each sample. The variability in the perturbed networks was captured in an augmented dataset, which was then used for an age classification task. We found that resampling brain networks across a series of such numerically perturbed outcomes led to improved performance in all tested classifiers, preprocessing strategies, and dimensionality reduction techniques. Importantly, we find that this benefit does not hinge on a large number of perturbations, suggesting that even minimally perturbing a dataset adds meaningful variance which can be captured in the subsequently designed models.

show abstract

Numerical Uncertainty in Analytical Pipelines Lead to Impactful Variability in Brain Networks

Cited by 7 publications

References 38 publications

Evaluating the reliability of human brain white matter tractometry

Evaluating the reliability of human brain white matter tractometry

Data Augmentation Through Monte Carlo Arithmetic Leads to More Generalizable Classification in Connectomics

Data Augmentation Through Monte Carlo Arithmetic Leads to More Generalizable Classification in Connectomics

Contact Info

Product

Resources

About