What is the test-retest reliability of common task-fMRI measures? New empirical evidence and a meta-analysis

Abstract

Identifying brain biomarkers of disease risk and treatment response is a growing priority in neuroscience.The ability to identify meaningful biomarkers is fundamentally limited by measurement reliability; measures that do not yield reliable values are unsuitable as biomarkers to predict clinical outcomes. Measuring brain activity using task-fMRI is a major focus of biomarker development; however, the reliability of task-fMRI has not been systematically evaluated. We present converging evidence demonstrating poor reliability of task-fMRI measures. First, a meta-analysis of 90 experiments with 1,088 participants reporting 1,146 ICCs for task-fMRI revealed poor overall reliability (mean ICC = .397).Second, the test-retest reliabilities of activity in a priori regions of interest across 11 commonly used fMRI tasks collected in the Human Connectome Proj ect and the Dunedin Longitudinal Study were poor (ICCs = .067 -.485). Collectively, these findings demonstrate that commonly used task-fMRI measures are not currently suitable for brain biomarker discovery or individual differences research in cognitive neuroscience (i.e., brain-behavior mapping). We review how this state of affairs came to be and consider several avenues for improving the reliability of task-fMRI. Significance StatementA biomarker with the potential to be useful in predicting clinical outcomes must yield values that are repeatable. We performed a comprehensive meta-analysis of the test-retest reliability of task-fMRI measures, which are widely adopted for biomarker discovery in neuroscience. We found that the meta-analytic reliability of task-fMRI was poor. We also investigated the reliability of many of the most commonly used task-fMRI measures in two datasets recently collected with cutting-edge scanners and methods. We found generally poor reliability for these task-fMRI measures. These findings indicate that many task-fMRI measures are not currently suitable for biomarker discovery or individual differences research in cognitive neuroscience.

Reliability of a functional magnetic resonance imaging task of emotional conflict in healthy participants

“…Our findings are consistent with previous reports (Chase et al, ; Fournier et al, ; Lipp et al, ; Nord et al, ) as well as a current meta‐analysis (Elliott et al, ). Elliott et al () reported an average ICC of 0.397 for unthresholded ICC estimates across a variety of task‐based fMRI studies. Additionally, Nord et al (), reported very low ICC values for three emotional processing tasks in regions activated by the tasks such as the ACC and the amygdala.…”

“…Nonetheless, reliable activation of key regions across repeated testing in healthy participants was not apparent. Our results are in line with the conclusions of a recent meta‐analyses and subsequent confirmatory findings of poor test–retest reliability in a variety of fMRI‐based tasks (Elliott et al, ). This suggests that the suitability of such a task for uncovering biomarkers of treatment response in any patient population using repeated measures is questionable, as any associations with treatment would have to be distinguishable from fluctuations in activation that appear to be inherent to the task.…”

“…However, our assessment of reliability from week 0 to week 2 did not substantially improve reliability (see supplementary information). This is consistent with conclusions from Elliott et al () who reported that the test–retest interval had little impact on reliability estimates. Furthermore, we collected data from multiple sites using different scanners, and employed considerable efforts to maintain high image quality.…”

“…They report an average ICC value of 0.397 across 90 studies in their meta‐analysis, which reflects poor reliability. Independent analyses of 11 commonly used tasks revealed ICCs of <0.3, again indicating rather poor test–retest reliability (Elliott et al, ). However, it also has to be noted that poor reliability may not necessarily render a measure unusable as a biomarker, as they may not automatically reflect changes in task performance but individual differences (Hedge, Powell, & Sumner, ).…”

Facet‐level analysis of the relations between personality and laboratory aggression

“…Of note, such concerns about the construct validity of behavioral measures are by no means limited to aggression tasks. Meta‐analytic reviews observe null‐to‐small relations between self‐report and task measures of impulsivity (e.g., Cyders & Coskunpinar, 2011), and recent evidence suggests poor psychometric properties of many functional neuroimaging tasks (Elliott et al, 2019) and behavioral measures of self‐regulation (Enkavi et al, 2019). This criticism, of course, does not excuse or nullify critiques of the CRTT, but it is important to note that the difficulty of integrating self‐report and behavioral measures is a field‐wide dilemma and that the current study found more consistent relations between self‐reported traits and putatively relevant tasks than have been found when using many other laboratory tasks.…”