Trial and error: a hierarchical modeling approach to test-retest assessment

Abstract

The concept of test-retest reliability (TRR) indexes the repeatability or consistency of a measurement across time. High TRR of measures is critical for any scientific study, specifically for the study of individual differences. Evidence of poor TRR of commonly used behavioral and functional neuroimaging tasks is mounting (e.g., Hedge et al., 2018; Elliot et al., 2020). These reports have called into question the adequacy of using even the most common, well-characterized cognitive tasks with robust population-level task effects, to measure individual differences. Here, we demonstrate the limitations of the intraclass correlation coefficient (ICC), the classical metric that captures TRR as a proportional variance ratio. Specifically, the ICC metric is limited when characterizing TRR of cognitive tasks that rely on many individual trials to repeatedly evoke a psychological state or behavior. We first examine when and why conventional ICCs underestimate TRR. Further, based on recent foundational work (Rouder and Haaf, 2019; Haines et al., 2020), we lay out a hierarchical framework that takes into account the data structure down to the trial level and estimates TRR as a correlation divorced from trial-level variability. As part of this process, we examine several modeling issues associated with the conventional ICC formulation and assess how different factors (e.g., trial and subject sample sizes, relative magnitude of cross-trial variability) impact TRR. We reference the tools of TRR and 3dLMEr for the community to apply these models to behavior and neuroimaging data.

Hyperbolic trade-off: the importance of balancing trial and subject sample sizes in neuroimaging

“…Furthermore, recent investigations show that the variability ratio R v appears often to be greater than 1 and up to 100. For example, the R v ranged from 10 to 70 for the contrast between congruent and incongruent conditions among 12 regions in a classic Flanker FMRI experiment (Chen et al, 2021). In a reward-distraction FMRI experiment, the R v value ranged from 5 to 80 among 11 regions (Chen et al, 2020).…”

“…Finally, there are also practical, non-statistical considerations involved, such as feasibility, costs, scanner availability, etc. Even though recent work has lead to better characterizations of data hierarchy (Westfall et al, 2017;Chen et al, 2020;Chen et al, 2021), challenges remain from both a modeling and computational perspective.…”

“…In a reward-distraction FMRI experiment, the R v value ranged from 5 to 80 among 11 regions (Chen et al, 2020). Even for behavioral data, which are likely significantly less noisy than neuroimaging data, the cross-trial variability is large, with R v between 3 and 11 for reaction time data in a reward-distraction experiment (Chen et al, 2020), cognitive inhibition tasks such as the Stroop, Simon and Flanker task, digit-distance and grating orientation tasks (Rouder et al, 2019;Chen et al, 2021).…”

“…Even in scenarios where trial-level effects are a research focus (e.g., machine learning), within-subject crosstrial variability has not been systematically investigated. Recent attempts to model FMRI task data on the trial-level (Chen et al, 2021) have revealed just how large the variance across trials within the same condition can be-namely, many times the magnitude of between-subjects variance. Traditional analysis pipelines aggregate trials into a single regressor (e.g., condition mean) per subject via condition-level modeling.…”

“…A high reliability of individual differences in a Flanker task experiment means that subjects with a larger inhibition effect relative to the population average are expected to show a similar inhibition pattern when the experiment is repeated. Due to their smaller effect size compared to population effects, subject-level effects and reliability are much more subtle and may require hundreds of trials to achieve a reasonable precision for reliability estimation (Chen et al, 2021).…”