Using generalizability theory and the ERP reliability analysis (ERA) toolbox for assessing test-retest reliability of ERP scores part 2: Application to food-based tasks and stimuli

Abstract

If an ERP score is to reflect a trait-like characteristic or indicate if an intervention had an effect over time, adequate internal consistency and test-retest reliability of that ERP score across multiple testing sessions must be established. The current paper is a companion paper to Clayson et al. (current issue) that applied generalizability theory formulas and the ERP Reliability Analysis (ERA) Toolbox to assess test-retest and internal consistency in a dataset of eventrelated brain potentials (ERPs) assessing food-related cognition. Although ERPs in response to food cues have been related to eating behaviors or assessed during a health intervention, the reliability of food-related ERPs generally has not been tested. Within the generalizability theory framework, we assessed the stability (cf., test-retest reliability) and equivalence (cf., internal consistency) of four commonly used food-related ERPs: the late positive potential (LPP), centroparietal P3, N2, and fronto-central P3. 132 participants (92 female) completed two testing sessions held two weeks apart. During the sessions, participants completed a passive food viewing task, a high-calorie go/no-go task, and a low-calorie go/no-go task in a counterbalanced fashion. Coefficients of equivalence for all ERPs were excellent (> .96). Coefficients of stability were moderate-to-low, with N2 scores on the low-calorie go/no-go task showing the highest testretest reliability (> .65) and fronto-central P3 scores on the high-calorie go/no-go task showing the lowest (.48). Results suggest the ERPs in the current dataset have high internal consistency and would be reliable in detecting individual differences, but their test-retest reliability is limited. Reliability of these ERPs may be improved with changes in task stimuli, task instructions, and study procedures.

Normal congruency sequence effects in psychopathology: A behavioral and electrophysiological examination using a confound‐minimized design

“…Generalizability theory was used to calculate dependability (ϕ), which is an estimate of internal consistency (Baldwin et al, 2015; Brennan, 2001, 2003, 2010; Carbine et al, 2021; Clayson, Baldwin, & Larson, 2021; Clayson, Carbine, Baldwin, et al, 2021; Shavelson & Webb, 1991), but participants were not excluded based on internal consistency estimates due to the emphasis on within‐person variance. Generalizability theory is a multifaceted framework for estimating psychometric reliability that is well suited for studies of ERPs.…”

The psychometric upgrade psychophysiology needs

“…Therefore, this G-theory analysis pinpointed person × occasion variance as the necessary target for improving test-retest reliability of N2 scores. This indicates a need to determine the facets that impact individual differences between sessions, and in the context of the Carbine et al (2021) study that could include controlling for levels of satiety, time of the testing session, and levels of fatigue across measurement occasions.…”

“…Generalizability theory (G theory), developed over 60 years ago (Brennan, 2001; Cronbach et al., 1963, 1972; Shavelson & Webb, 1991; Vispoel et al., 2018a, 2018b), is a measurement theory with a flexible and multifaceted framework that is well suited to handling the data structures typical of ERP scores. The application of G theory to ERP research has been extensively described elsewhere (Baldwin et al., 2015; Carbine et al., 2021; Clayson, Baldwin, & Larson, 2021; Clayson, Brush, & Hajcak, 2021; Clayson, Carbine, Baldwin, et al., 2021; Clayson & Miller, 2017a, 2017b). Considering that classical test theory (CT theory) remains popular among psychophysiologists, CT theory will be briefly described, then a brief overview of important aspects of G theory is provided, and the two measurement theories are contrasted with an eye toward application to studies of psychophysiology.…”

“…The following example illustrates how G theory can be used to identify the key sources of error variance that limit the reliability of scores. G theory was used to examine the internal consistency (coefficient of equivalence) and test–retest reliability (coefficient of stability) of N2 ERP component scores following food‐related go and nogo stimuli (Carbine et al., 2021; Clayson, Carbine, Baldwin, et al., 2021). The estimation of reliability used a two‐facet model that included items and measurement occasion, allowing the partitioning of observed score variance into person‐related, item‐related, and occasion‐related variances (see Figure 2 for Venn diagram).…”

The effects of daily step goals of 10,000, 12,500, and 15,000 steps per day on neural activity to food cues: A 24‐week dose‐response randomized trial

“…The LPP is also related to emotional reactivity and emotional regulation (Hajcak et al, 2014), with a growing consensus that the LPP represents an automatic sustained engagement to content with strong emotional significance (Hajcak & Foti, 2020). In the context of food-related picture viewing, LPP amplitude has a larger sustained response to food stimuli relative to neutral and, similar to P3 amplitude, to high-calorie stimuli relative to low-calorie stimuli (Carbine et al, 2021(Carbine et al, , 2018Chami et al, 2019;Sarlo et al, 2013).…”