Comparing perturbation models for evaluating stability of neuroimaging pipelines

Kiar, Gregory; Castro, Pablo de Oliveira; Rioux, Pierre; Petit, Éric; Brown, Shawn T.; Evans, Alan C.; Glatard, Tristan

doi:10.1177/1094342020926237

Cited by 22 publications

(25 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Both the RR and PB variants of MCA were used independently for all experiments. As was presented in 4 , both the degree of instrumentation (i.e. number of affected libraries) and the perturbation mode have an effect on the distribution of observed results.…”

Section: Methodsmentioning

confidence: 78%

“…Graphs, in the form of adjacency matrices, were compared to one another using three metrics: normalized percent deviation, Pearson correlation, and edgewise significant digits. The normalized percent deviation measure, defined in 4 , scales the norm of the difference between a simulated graph and the reference execution (that without intentional perturbation) with respect to the norm of the reference graph. The purpose of this comparison is to provide insight on the scale of differences in observed graphs relative to the original signal intensity.…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Numerical Uncertainty in Analytical Pipelines Lead to Impactful Variability in Brain Networks

Kiar

Chatelain

Castro

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

The analysis of brain-imaging data requires complex and often non-linear transformations to support findings on brain function or pathologies. And yet, recent work has shown that variability in the choices that one makes when analyzing data can lead to quantitatively and qualitatively different results, endangering the trust in conclusions. Even within a given method or analytical technique, numerical instabilities could compromise findings. We instrumented a structural-connectome estimation pipeline with Monte Carlo Arithmetic, a technique to introduce random noise in floating-point computations, and evaluated the stability of the derived connectomes, their features, and the impact on a downstream analysis. The stability of results was found to be highly dependent upon which features of the connectomes were evaluated, and ranged from perfectly stable (i.e. no observed variability across executions) to highly unstable (i.e. the results contained no trustworthy significant information). While the extreme range and variability in results presented here could severely hamper our understanding of brain organization in brain-imaging studies, it also leads to an increase in the reliability of datasets. This paper highlights the potential of leveraging the induced variance in estimates of brain connectivity to reduce the bias in networks alongside increasing the robustness of their applications in the detection or classification of individual differences. This paper demonstrates that stability evaluations are necessary for understanding error and bias inherent to scientific computing, and that they should be a component of typical analytical workflows.

show abstract

Section: Methodsmentioning

confidence: 78%

Section: Methodsmentioning

confidence: 99%

Numerical Uncertainty in Analytical Pipelines Lead to Impactful Variability in Brain Networks

Kiar

Chatelain

Castro

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…It would be relevant to investigate whether the observed instability of registration processes generalizes to other toolkits or remains specific to FSL. In view of the effect of small data perturbations in a variety of toolboxes and processes, such as cortical surface extraction using FreeSurfer and CIVET [ 12 ] or connectome estimation using Dipy [ 31 ], it is probable that this observation generalizes widely across toolboxes and requires a deeper investigation of the stability of linear and non-linear registration.…”

Section: Discussionmentioning

confidence: 99%

“…A possible method to study this problem more comprehensively would be to introduce controlled numerical perturbations in pipelines, which could be done by introducing noise either in the data, or in floating-point computations through MCA [ 15 ]. Beauzamy [ 31 ] discusses and compares these 2 techniques.…”

Section: Discussionmentioning

confidence: 99%

File-based localization of numerical perturbations in data analysis pipelines

et al. 2020

Self Cite

View full text Add to dashboard Cite

Background Data analysis pipelines are known to be affected by computational conditions, presumably owing to the creation and propagation of numerical errors. While this process could play a major role in the current reproducibility crisis, the precise causes of such instabilities and the path along which they propagate in pipelines are unclear. Method We present Spot, a tool to identify which processes in a pipeline create numerical differences when executed in different computational conditions. Spot leverages system-call interception through ReproZip to reconstruct and compare provenance graphs without pipeline instrumentation. Results By applying Spot to the structural pre-processing pipelines of the Human Connectome Project, we found that linear and non-linear registration are the cause of most numerical instabilities in these pipelines, which confirms previous findings.

show abstract

“…Perturbation methods which inject small amounts of noise through the execution of a pipeline, such as Monte Carlo Arithmetic (MCA) 17,18 , have recently been used to induce instabilities in structural connectome estimation software 19 .…”

Section: Introductionmentioning

confidence: 99%

Data Augmentation Through Monte Carlo Arithmetic Leads to More Generalizable Classification in Connectomics

Kiar

Chatelain

Salari

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Machine learning models are commonly applied to human brain imaging datasets in an effort to associate function or structure with behaviour, health, or other individual phenotypes. Such models often rely on low-dimensional maps generated by complex processing pipelines. However, the numerical instabilities inherent to pipelines limit the fidelity of these maps and introduce computational bias. Monte Carlo Arithmetic, a technique for introducing controlled amounts of numerical noise, was used to perturb a structural connectome estimation pipeline, ultimately producing a range of plausible networks for each sample. The variability in the perturbed networks was captured in an augmented dataset, which was then used for an age classification task. We found that resampling brain networks across a series of such numerically perturbed outcomes led to improved performance in all tested classifiers, preprocessing strategies, and dimensionality reduction techniques. Importantly, we find that this benefit does not hinge on a large number of perturbations, suggesting that even minimally perturbing a dataset adds meaningful variance which can be captured in the subsequently designed models.

show abstract

Comparing perturbation models for evaluating stability of neuroimaging pipelines

Cited by 22 publications

References 15 publications

Numerical Uncertainty in Analytical Pipelines Lead to Impactful Variability in Brain Networks

Numerical Uncertainty in Analytical Pipelines Lead to Impactful Variability in Brain Networks

File-based localization of numerical perturbations in data analysis pipelines

Data Augmentation Through Monte Carlo Arithmetic Leads to More Generalizable Classification in Connectomics

Contact Info

Product

Resources

About