Robust group- but limited individual-level (longitudinal) reliability and insights into cross-phases response prediction of conditioned fear

Abstract

Here we follow the call to target measurement reliability as a key prerequisite for individual-level predictions in translational neuroscience by investigating i) longitudinal reliability at the individual and ii) group level, iii) internal consistency and iv) response predictability across experimental phases. 120 individuals performed a fear conditioning paradigm twice six months apart. Analyses of skin conductance responses, fear ratings and BOLD-fMRI with different data transformations and included numbers of trials were conducted. While longitudinal reliability was rather limited at the individual level, it was comparatively higher for acquisition but not extinction at the group-level. Internal consistency was satisfactory. Higher responding in preceding phases predicted higher responding in subsequent experimental phases at a weak to moderate level depending on data specifications. In sum, the results suggest that while individual-level predictions are meaningful for (very) short time frames, they also call for more attention to measurement properties in the field.

Test–retest reliability of human threat conditioning and generalization across a 1‐to‐2‐week interval

“…In the current study, we use the most common SCR approach in the generalization literature to maximize the compatibility of our work with the prior literature. That said, the field would benefit from more formal analysis of different physiological quantification pipelines in relation to generalization reliability (in line with a move toward multiverse analyses, e.g., Klingelhöfer-Jens et al, 2022;Kuhn et al, 2022). One particularly important avenue for future research in this area are quantification approaches that minimize trial-by-trial variability (e.g., model-based approaches, see Kuhn et al, 2022), and therefore would potentially limit the impact that initial learning trials during the first session have on overall test-retest reliability.…”

Enhancing Precision in Human Neuroscience

“…Few of these steps have been systematically investigated with respect to measurement precision. Recent multiverse-type work suggests that effect sizes and precision derived from different processing and operationalization steps differ substantially despite identical underlying data (Klingelhöfer-Jens et al, 2022;Pineles et al, 2009;Sjouwerman et al, 2016). Furthermore, exclusion of participants due to non-responding in SCRs is based on heterogeneous definitions with potential consequences on measurement reliability and precision (Lonsdorf et al, 2019).…”

“…Allen & Yen, 2001) is not straightforward for SCRs. Indeed, larger trial numbers did not generally improve reliability estimates of SCRs (in a learning paradigm; Klingelhöfer-Jens et al, 2022). One interpretation of this result is that increasing precision by aggregation over more trials can get counteracted by sequence effects.…”

“…For research questions targeting group differences, group-level precision should be investigated; while for correlational hypotheses, subject-level precision is the outcome of choice (for an example on reliability, see Parsons, 2020). As little is known about the potentially differential precision of different outcome measures, it is important to address this question across various measures and paradigms (Klingelhöfer-Jens et al, 2022). For instance, the complex relationship between trial number and precision should be further evaluated across different outcome measures as sequence effects complicate the influence of increasing trial number on precision estimates due to habituation effects and fatigue counteracting the merit of additional observations (see e.g., EDA vs. EEG).…”

Improving the Reliability of Cognitive Task Measures: A Narrative Review

“…triallevel variance), and n is the number of trials. In practice, this relationship often holds [8,9] though with some exceptions [51,52]. Notably, increasing the number of task trials only benefits reliability if measurement error is random.…”