Addressing the reliability fallacy in fMRI: Similar group effects may arise from unreliable individual effects

Fröhner, Juliane H.; Teckentrup, Vanessa; Smolka, Michael N.; Kroemer, Nils B.

doi:10.1016/j.neuroimage.2019.03.053

Cited by 87 publications

(96 citation statements)

References 80 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Nevertheless, these studies, including the current one, are based on group inference. The applications of olfactory fMRI in clinical diagnosis requires more intra‐ and inter‐ individual research to have a reliable and strong activation pattern in response to odor stimulation, given the low reliability of fMRI results …”

Section: Discussionmentioning

confidence: 99%

Short or long runs: An exploratory study of odor‐induced fMRI design

et al. 2019

View full text Add to dashboard Cite

Objective: Functional magnetic resonance imaging (fMRI) is a non-invasive neuroimaging technique widely used in olfactory research. During a typical fMRI olfactory block-design, one functional "run" refers to a combination of multiple blocks with continuous brain image acquisition. The current study investigated the length of functional runs on odor-induced brain response signals (blood oxygen level dependent [BLOD]) within the primary and key secondary olfactory areas.Methods: Twenty-five female adults (age range 19 to 30 years, mean age 25 years) underwent a block-design fMRI measurement with odor stimulation. Twelve participants received the odor stimuli within a short run paradigm (six blocks in each 4-minute run, eight runs in total), and 13 participants received the odor stimulation with a long-run paradigm (12 blocks in each 8-minute run, four runs in total). For each paradigm, two odors (peach and rose) were alternatingly presented between runs. Participants rated odor intensity and pleasantness at the end of each run. Ratings and fMRI data were analyzed for different subsections and compared between groups.Results: There was a higher level of brain activation in the insula and orbitofrontal cortex during the short-run paradigm as compared to the long-run paradigm. However, there was no difference for odor intensity or pleasantness ratings.Conclusion: The current study suggested the employment of short runs with multiple repetitions for odor stimulation during fMRI research.

show abstract

Section: Discussionmentioning

confidence: 99%

Short or long runs: An exploratory study of odor‐induced fMRI design

et al. 2019

View full text Add to dashboard Cite

show abstract

“…Accordingly, there exists no routinely used clinical application of fMRI outside of the field of presurgical mapping due to the lack of sufficient reliability in the measurement of individual activation strength-for example task, instruction, and different levels of attention may influence reliability (45,46). Group studies, by contrast, show a much higher reliability in detecting deviations in activation strength (47). Therefore, almost exclusively all fMRI studies in psychiatry have explored cohorts of patients (see Table 1).…”

Section: Current Clinical Applications Of Fmrimentioning

confidence: 99%

Current Challenges in Translational and Clinical fMRI and Future Directions

Specht

2020

Front. Psychiatry

View full text Add to dashboard Cite

Translational neuroscience is an important field that brings together clinical praxis with neuroscience methods. In this review article, the focus will be on functional neuroimaging (fMRI) and its applicability in clinical fMRI studies. In the light of the "replication crisis," three aspects will be critically discussed: First, the fMRI signal itself, second, current fMRI praxis, and, third, the next generation of analysis strategies. Current attempts such as restingstate fMRI, meta-analyses, and machine learning will be discussed with their advantages and potential pitfalls and disadvantages. One major concern is that the fMRI signal shows substantial within-and between-subject variability, which affects the reliability of both taskrelated, but in particularly resting-state fMRI studies. Furthermore, the lack of standardized acquisition and analysis methods hinders the further development of clinical relevant approaches. However, meta-analyses and machine-learning approaches may help to overcome current shortcomings in the methods by identifying new, and yet hidden relationships, and may help to build new models on disorder mechanisms. Furthermore, better control of parameters that may have an influence on the fMRI signal and that can easily be controlled for, like blood pressure, heart rate, diet, time of day, might improve reliability substantially.

show abstract

“…Within-subject robustness is, then, often inappropriately invoked to suggest between-subject reliability, despite the fact that reliable within-subject experimental effects at a group level can arise from unreliable between-subjects measurements (30) .…”

Section: The Two Disciplines Of Fmri Researchmentioning

confidence: 99%

What is the test-retest reliability of common task-fMRI measures? New empirical evidence and a meta-analysis

Elliott

Knodt

Ireland

et al. 2019

Preprint

View full text Add to dashboard Cite

Identifying brain biomarkers of disease risk and treatment response is a growing priority in neuroscience.The ability to identify meaningful biomarkers is fundamentally limited by measurement reliability; measures that do not yield reliable values are unsuitable as biomarkers to predict clinical outcomes. Measuring brain activity using task-fMRI is a major focus of biomarker development; however, the reliability of task-fMRI has not been systematically evaluated. We present converging evidence demonstrating poor reliability of task-fMRI measures. First, a meta-analysis of 90 experiments with 1,088 participants reporting 1,146 ICCs for task-fMRI revealed poor overall reliability (mean ICC = .397).Second, the test-retest reliabilities of activity in a priori regions of interest across 11 commonly used fMRI tasks collected in the Human Connectome Proj ect and the Dunedin Longitudinal Study were poor (ICCs = .067 -.485). Collectively, these findings demonstrate that commonly used task-fMRI measures are not currently suitable for brain biomarker discovery or individual differences research in cognitive neuroscience (i.e., brain-behavior mapping). We review how this state of affairs came to be and consider several avenues for improving the reliability of task-fMRI. Significance StatementA biomarker with the potential to be useful in predicting clinical outcomes must yield values that are repeatable. We performed a comprehensive meta-analysis of the test-retest reliability of task-fMRI measures, which are widely adopted for biomarker discovery in neuroscience. We found that the meta-analytic reliability of task-fMRI was poor. We also investigated the reliability of many of the most commonly used task-fMRI measures in two datasets recently collected with cutting-edge scanners and methods. We found generally poor reliability for these task-fMRI measures. These findings indicate that many task-fMRI measures are not currently suitable for biomarker discovery or individual differences research in cognitive neuroscience.

show abstract

Addressing the reliability fallacy in fMRI: Similar group effects may arise from unreliable individual effects

Cited by 87 publications

References 80 publications

Short or long runs: An exploratory study of odor‐induced fMRI design

Short or long runs: An exploratory study of odor‐induced fMRI design

Current Challenges in Translational and Clinical fMRI and Future Directions

What is the test-retest reliability of common task-fMRI measures? New empirical evidence and a meta-analysis

Contact Info

Product

Resources

About