Multisite Comparison of MRI Defacing Software Across Multiple Cohorts

Abstract

With improvements to both scan quality and facial recognition software, there is an increased risk of participants being identified by a 3D render of their structural neuroimaging scans, even when all other personal information has been removed. To prevent this, facial features should be removed before data are shared or openly released, but while there are several publicly available software algorithms to do this, there has been no comprehensive review of their accuracy within the general population. To address this, we tested multiple algorithms on 300 scans from three neuroscience research projects, funded in part by the Ontario Brain Institute, to cover a wide range of ages (3–85 years) and multiple patient cohorts. While skull stripping is more thorough at removing identifiable features, we focused mainly on defacing software, as skull stripping also removes potentially useful information, which may be required for future analyses. We tested six publicly available algorithms (afni_refacer, deepdefacer, mri_deface, mridefacer, pydeface, quickshear), with one skull stripper (FreeSurfer) included for comparison. Accuracy was measured through a pass/fail system with two criteria; one, that all facial features had been removed and two, that no brain tissue was removed in the process. A subset of defaced scans were also run through several preprocessing pipelines to ensure that none of the algorithms would alter the resulting outputs. We found that the success rates varied strongly between defacers, with afni_refacer (89%) and pydeface (83%) having the highest rates, overall. In both cases, the primary source of failure came from a single dataset that the defacer appeared to struggle with - the youngest cohort (3–20 years) for afni_refacer and the oldest (44–85 years) for pydeface, demonstrating that defacer performance not only depends on the data provided, but that this effect varies between algorithms. While there were some very minor differences between the preprocessing results for defaced and original scans, none of these were significant and were within the range of variation between using different NIfTI converters, or using raw DICOM files.

Segmentation stability of human head and neck cancer medical images for radiotherapy applications under de-identification conditions: benchmarking data sharing and artificial intelligence use-cases

“…As we demonstrate on both pooled analysis and investigation of individual OARs for auto-segmentation model training and evaluation, performance is often modestly decreased for the fsl_deface and greatly decreased for pydeface. While pydeface is often shown to have favorable outcomes for facial anonymization as shown in previous studies 14,35 , the trade-off for certain applications, such as radiotherapy imaging, becomes apparent. Therefore, in cases where defacing is unavoidable, fsl_deface should likely be preferred for radiotherapy segmentation applications.…”

Impact of defacing on automated brain atrophy estimation

“…Recently, machine learning-based face recognition approaches have been shown to be alarmingly successful in matching photographs of participants to their respective MRI scans, with a success rate of up to 97% [ 2 , 3 ]. On the other hand, concerns have been raised on the data integrity after defacing, especially with regard to common volumetric analysis [ 13 , 14 ], while other studies have shown modest to no effects of defacing on common neuroscientific analysis pipelines [ 3 , 15 ].…”

“…The most popular defacing approaches include afni_refacer (based on AFNI [ 7 ]), mask_face (released by the Neuroinformatics Research Group [ 8 ]), mri_deface (based on FreeSurfer [ 9 ]), fsl_deface (based on FMRIB’s Software Library (FSL) [ 10 ]), PyDeface (released by the Poldrack Lab [ 11 ]) and spm_deface (included in the Statistical Parametric Mapping (SPM) package [ 12 ]). However, concerns have been raised with respect to data alteration after defacing, with some studies reporting significant deviations in brain volume assessments [ 13 , 14 ], while other studies have shown almost no effects of defacing [ 3 , 15 ].…”

Defacing biases manual and automated quality assessments of structural MRI with MRIQC

“…T1w image before and after defacing (adapted from (Theyers et al 2021)) B) Eye spillover is an example of information used for rating image quality that is removed by defacing. C) The Bland-Altman plot shows the evolution of the rating before vs after defacing.…”