Assessing the reliability of single-item momentary affective measurements in experience sampling.

Abstract

Emotion researchers that use experience sampling methods (ESM) study how emotions fluctuate in everyday life. To reach valid conclusions, confirming the reliability of momentary emotion measurements is essential. However, to minimize participant burden, ESM researchers often use single-item measures, preventing a reliability assessment of people’s emotion ratings. Furthermore, because emotions constantly change, checking reliability via conventional test–retest procedures is impractical, for it is impossible to separate measurement error from meaningful emotional variability. Here, drawing from classical test theory (CTT), we propose two time-varying test–retest adaptations to evaluate the reliability of single-item (emotion) measures in ESM. Following Method 1, we randomly repeat one emotion item within the same momentary survey and evaluate the discrepancy between test and retest ratings to determine reliability. Following Method 2, we introduce a subsequent, shortly delayed retest survey and extrapolate the size of test–retest discrepancies to the hypothetical instance where no time between assessments would exist. First, in an analytical study, we establish the mathematical relation between observed test–retest discrepancies and measurement error variance for both methods, based on common assumptions in the CTT literature. Second, in two empirical studies, we apply both methods to real-life emotion time series and find that the size of error in people’s emotion ratings corresponds with almost a 10th of the scale, comprising around 27% of the total variability in participants’ affective responses. Consequently, disregarding measurement error in ESM is problematic, and we encourage researchers to include a test–retest procedure in their future studies when relying on single-item measures.

For better or for worse? Visualizing previous intensity levels improves emotion (dynamic) measurement in experience sampling.

“…In this regard, a common criticism of the latent AR(1) model is that it is too basic to accurately represent the complex nature of our emotional system (Loossens et al, 2020). This generally leads to an overestimation of measurement error variance (Dejonckheere et al, 2022), but it remains unclear whether different types of measurement are affected to a similar extent. Relatedly, when model assumptions of the latent AR(1) model are severely violated (e.g., nonstationarity in REL time series; Bringmann et al, 2013), the biased parameters will not produce a reliable estimate of the measurement error variance in people’s ESM time series.…”

“…When a series of emotion ratings does not correctly represent the true natural fluctuations in people's emotions and intraindividual changes in observed emotion ratings are mainly driven by random fluctuations caused by measurement error, the true dynamic qualities of people's emotions are not revealed (Schuurman et al, 2015). Likewise, detecting meaningful within-person associations with other momentary constructs will be harder because noisy ratings are known to attenuate statistical power at the within-person level (Dejonckheere et al, 2022). Therefore, it is important to determine the internal validity of an emotion time series, which refers to the extent with which we can accurately predict the true future dynamic trajectory of an individual's emotional state based on previous emotional assessments.…”

Measuring affect in daily life: A multilevel psychometric evaluation of the PANAS-X across four ecological momentary assessment samples.

“…For example, the equation for R C (Equation 5) does not include item-level variance estimates and would be interpreted in a similar fashion for a single-item measure. However, examinations of single-item reliability suggest that using more than one item to measure a construct may be beneficial, as single items may be associated with greater error variance (Dejonckheere et al, 2022).…”

Unlocking nonlinear dynamics and multistability from intensive longitudinal data: A novel method.

“…For example, the animal movement data in ecology used by Brillinger (2007) has 1,571 data points measured by the Global Positioning System with a precision in the meter level and range of kilometers, and the long-term geographical data used by Livina et al (2010) has 3,000 data points of isotope concentration which could also be measured precisely comparing with the range of fluctuation. To compare with, experience sampling method data sets often contain around a hundred data points, with the measurement noise comprising about a quarter of the total variation (Dejonckheere et al, 2022).…”