The art and science of using quality control to understand and improve fMRI data

Abstract

Designing and executing a good quality control (QC) process is vital to robust and reproducible science and is often taught through hands on training. As FMRI research trends toward studies with larger sample sizes and highly automated processing pipelines, the people who analyze data are often distinct from those who collect and preprocess the data. While there are good reasons for this trend, it also means that important information about how data were acquired, and their quality, may be missed by those working at later stages of these workflows. Similarly, an abundance of publicly available datasets, where people (not always correctly) assume others already validated data quality, makes it easier for trainees to advance in the field without learning how to identify problematic data. This manuscript is designed as an introduction for researchers who are already familiar with fMRI, but who did not get hands on QC training or who want to think more deeply about QC. This could be someone who has analyzed fMRI data but is planning to personally acquire data for the first time, or someone who regularly uses openly shared data and wants to learn how to better assess data quality. We describe why good QC processes are important, explain key priorities and steps for fMRI QC, and as part of the FMRI Open QC Project, we demonstrate some of these steps by using AFNI software and AFNI’s QC reports on an openly shared dataset. A good QC process is context dependent and should address whether data have the potential to answer a scientific question, whether any variation in the data has the potential to skew or hide key results, and whether any problems can potentially be addressed through changes in acquisition or data processing. Automated metrics are essential and can often highlight a possible problem, but human interpretation at every stage of a study is vital for understanding causes and potential solutions.

A Set of FMRI Quality Control Tools in AFNI: Systematic, in-depth and interactive QC with afni_proc.py and more

“…This has been shown to be useful in several cases, even in widely used public repositories such as ABIDE, OpenNeuro and the Functional Connectome Project. The images help validate whether the estimated cost function comparison matches with the data (i.e., the cortical patterns of the chosen “correct” answer really do match better than in the other case); in fact, using AFNI’s typical local Pearson correlation (LPC; Saad et al, 2009) for this process, left-right flip guesses for human data tend to be highly accurate and were even useful in identifying a dataset whose EPI and anatomical appeared to erroneously belong to different subjects (Reynolds et al, 2023; Teves et al, 2023).…”

“…The process of performing quality control on data should encompass more than simple box-ticking of "include" or "exclude" a subject, and instead lead to much more complete understanding of the data in all of its stages from collection through intermediate (pre)processing to final analyses (Reynolds et al, 2023). As such, it should be actively planned during the study design phase (Teves et al, 2023), and some measures can be gathered and assessed even during the acquisition time (Etzel, 2023). QC checks remain necessary for evaluating the suitability of the data for the research at hand, and they should be integrated early on in data collection-preferably in a piloting stage.…”

“…More blocks and items will likely be added over time. For a more complete list of QC items to consider evaluating within each block, see the main text here as well as Reynolds et al (2023), Birn (2023), Lepping et al (2023) and Teves et al (2023).…”

“…The aim of this arrangement is to simplify recording and reporting of the QC itself. The APQC HTML was shown to be helpful in practice in evaluating data quality within the FMRI Open QC Project (see Reynolds et al, 2023; Teves et al, 2023; Lepping et al 2023; Birn, 2023).…”

Submillimeter fMRI Acquisition Techniques for Detection of Laminar and Columnar Level Brain Activation

“…In general, the quality control (QC) procedure for fMRI requires various assessment tasks to be performed, depending on the purpose of the study and the stage of the experiment. 103 This sub-section describes QC criteria that directly relate to the assessment of submillimeter fMRI data.…”