Inter-rater reliability of functional MRI data quality control assessments: A standardised protocol and practical guide using pyfMRIqc

Williams, B. A.; Hedger, Nicholas; McNabb, Carolyn Beth; Rossetti, Gabriella; Christakou, Anastasia

doi:10.3389/fnins.2023.1070413

Cited by 5 publications

(6 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This is particularly the case when images are made systematically and can be compared easily across subjects (Lepping et al, 2023). Integrated qualitative-quantitative reviews also usefully inform discussions of inter-rater variability, which necessarily exist in data assessment (Williams et al, 2023): consensus building becomes easier during the training phase, and differences of interpretation are more readily discussed. The APQC HTML usefully contributes to all of these aspects.…”

Section: Discussionmentioning

confidence: 99%

“…There are too many aspects to compare across software packages and projects. We refer people instead to the documentation of the individual packages, as well as to the useful examples in the demonstration papers for those participated in the FMRI Open QC Project: AFNI (Birn, 2023); SPM (Di and Biswal, 2023); R scripts and AFNI (Etzel, 2023); AFNI (Lepping et al, 2023); DPABISurf, DPARF and fMRIPrep (Lu and Yan, 2023); CONN, SPM12 and FSLeyes (Morfini et al, 2023); MRIQC and fMRIPrep (Provins et al, 2023); AFNI (Reynolds et al, 2023); AFNI (Teves et al, 2023); and pyfMRIqc (Williams et al, 2023). These provide useful QC tutorials on many of these packages, with lists of specific QC items, detailed descriptions, and example cases.…”

Section: Qc Tools Across the Fieldmentioning

confidence: 99%

“…In contrast, Hagler et al (2019) described a combination of automated quantitative summary parameters and manual QC, such as using montages of images, in the ABCD study. Monereo-Sanchez (2021) described a comparison of QC procedures for a relatively large sampling of structural datasets from the Maastricht Study, concluding that those including manual/visual QC resulted in the least unexplained variance; however, they also noted this required training to reduce bias or subjectivity (and for related comments on FMRI specifically, see Williams et al, 2023). Ma et al (2022) applied outlier detection to both the multimodal UK Biobank and Human Connectome Project data collections, followed by visual assessment of those extrema.…”

Section: The Need For (More) Qc In Fmri Researchmentioning

confidence: 99%

See 2 more Smart Citations

A Set of FMRI Quality Control Tools in AFNI: Systematic, in-depth and interactive QC with afni_proc.py and more

Taylor,

Glen,

Chen

et al. 2024

Preprint

View full text Add to dashboard Cite

Quality control (QC) assessment is a vital part of FMRI processing and analysis, and a typically under-discussed aspect of reproducibility. This includes checking datasets at their very earliest stages (acquisition and conversion) through their processing steps (e.g., alignment and motion correction) to regression modeling (correct stimuli, no collinearity, valid fits, enough degrees of freedom, etc.) for each subject. There are a wide variety of features to verify throughout any single subject processing pipeline, both quantitatively and qualitatively. We present several FMRI preprocessing QC features available in the AFNI toolbox, many of which are automatically generated by the pipeline-creation tool, afni_proc.py. These items include: a modular HTML document that covers full single subject processing from the raw data through statistical modeling; several review scripts in the results directory of processed data; and command line tools for identifying subjects with one or more quantitative properties across a group (such as triaging warnings, making exclusion criteria or creating informational tables). The HTML itself contains several buttons that efficiently facilitate interactive investigations into the data, when deeper checks are needed beyond the systematic images. The pages are linkable, so that users can evaluate individual items across group, for increased sensitivity to differences (e.g., in alignment or regression modeling images). Finally, the QC document contains rating buttons for each "QC block", as well as comment fields for each, to facilitate both saving and sharing the evaluations. This increases the specificity of QC, as well as its shareability, as these files can be shared with others and potentially uploaded into repositories, promoting transparency and open science. We describe the features and applications of these QC tools for FMRI.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Qc Tools Across the Fieldmentioning

confidence: 99%

Section: The Need For (More) Qc In Fmri Researchmentioning

confidence: 99%

See 1 more Smart Citation

A Set of FMRI Quality Control Tools in AFNI: Systematic, in-depth and interactive QC with afni_proc.py and more

Taylor,

Glen,

Chen

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…The authors found that the inter-rater agreement varied depending on the stage of the disease, with higher agreement generally observed between experts than between novices, particularly for non-glioblastoma cases. A different study presents a quality control protocol using an automated tool for assessing functional MRI data quality and assesses the inter-rater reliability of four independent raters [30]. The authors suggest several approaches to increase rater agreement and reduce disagreement for uncertain cases, ultimately aiming to improve classification consistency in data quality assessments.…”

Section: Approaches To Label Quality Assurancementioning

confidence: 99%

Machine Learning-Based Label Quality Assurance for Object Detection Projects in Requirements Engineering

Pičuljan

Car

2023

Applied Sciences

View full text Add to dashboard Cite

In recent years, the field of artificial intelligence has experienced significant growth, which has been primarily attributed to advancements in hardware and the efficient training of deep neural networks on graphics processing units. The development of high-quality artificial intelligence solutions necessitates a strong emphasis on data-centric approaches that involve the collection, labeling and quality-assurance of data and labels. These processes, however, are labor-intensive and often demand extensive human effort. Simultaneously, there exists an abundance of untapped data that could potentially be utilized to train models capable of addressing complex problems. These raw data, nevertheless, require refinement to become suitable for machine learning training. This study concentrates on the computer vision subdomain within artificial intelligence and explores data requirements within the context of requirements engineering. Among the various data requirement activities, label quality assurance is crucial. To address this problem, we propose a machine learning-based method for automatic label quality assurance, especially in the context of object detection use cases. Our approach aims to support both annotators and computer vision project stakeholders while reducing the time and resources needed to conduct label quality assurance activities. In our experiments, we trained a neural network on a small set of labeled data and achieved an accuracy of 82% in differentiating good and bad labels on a large set of labeled data. This demonstrates the potential of our approach in automating label quality assurance.

show abstract

“…For these cases and for the tasks where an automatic tool is not available (e.g., ROI segmentations performed outside FreeSurfer, inspection for incidental findings), manual curation is still the most reliable approach to QC. 20 QA/QC protocols for the manual inspection of structural and functional MRI data have also been developed using a variety of tools like Freesurfer, 21 pyfMRIQC 22 and MRIQC together with fMRIprep. 12 Even with the most detailed protocols, the usual strategy when doing visual inspection involves opening the QA tool or the generated report and logging the assessment in a separate file, conventionally a spreadsheet, repeating the process for each case to evaluate.…”

Section: Introductionmentioning

confidence: 99%

QRATER: a collaborative and centralized imaging quality control web-based application

Fernandez-Lozano,

Dadar,

Morrison

et al. 2024

Aperture Neuro

View full text Add to dashboard Cite

Quality control (QC) is an important part of all scientific analyses, including neuroscience. With manual curation considered the gold standard, there remains a lack of available tools that make manual neuroimaging QC accessible, fast, and easy. In this article we present Qrater, a containerized web-based Python application that enables viewing and rating any type of image for QC purposes. Qrater functionalities allow collaboration between various raters on the same dataset which can facilitate completing large QC tasks. Qrater was used to evaluate QC rater performance on three different magnetic resonance (MR) image QC tasks by a group of raters having different amounts of experience. The tasks included QC of raw MR images (10,196 images), QC of linear registration to a standard template (10,196 images), and QC of skull segmentation (6,968 images). We measured the proportion of failed images, average rating time per image, intra- and inter-rater agreement, as well as the comparison against QC using a conventional method. The median time spent rating per image differed significantly between raters (depending on rater experience) in each of the three QC tasks. Evaluating raw MR images was slightly faster using Qrater than an image viewer (expert: 99 vs. 90 images in 63 min; trainee 99 vs 79 images in 98 min). Reviewing the linear registration using Qrater was twice faster for the expert (99 vs. 43 images in 36 min) and three times faster for the trainee (99 vs. 30 images in 37 min). The greatest difference in rating speed resulted from the skull segmentation task where the expert took a full minute to inspect the volume on a slice-by-slice basis compared to just 3 s using Qrater. Rating agreement also depended on the experience of the raters and the task at hand: trained raters’ inter-rater agreements with the expert’s gold standard were moderate for both raw images (Fleiss’ Kappa = 0.44) and linear registration (Fleiss’ Kappa = 0.56); the experts’ inter-rater agreement of the skull segmentation task was excellent (Cohen’s Kappa = 0.83). These results demonstrate that Qrater is a useful asset for QC tasks that rely on manual evaluation of QC images.

show abstract

Inter-rater reliability of functional MRI data quality control assessments: A standardised protocol and practical guide using pyfMRIqc

Cited by 5 publications

References 44 publications

A Set of FMRI Quality Control Tools in AFNI: Systematic, in-depth and interactive QC with afni_proc.py and more

A Set of FMRI Quality Control Tools in AFNI: Systematic, in-depth and interactive QC with afni_proc.py and more

Machine Learning-Based Label Quality Assurance for Object Detection Projects in Requirements Engineering

QRATER: a collaborative and centralized imaging quality control web-based application

Contact Info

Product

Resources

About