Identification of careless responding in ecological momentary assessment research: From posthoc analyses to real-time data monitoring.

Abstract

With the emerging ubiquity of cell phones, ecological momentary assessment (EMA) as a set of methods enable researchers to study momentary social, psychological, and affective responses to everyday life. Additionally, EMA enables researchers to acquire longitudinal data without the need for multiple lab visits. As the use of EMA in research increases, so too does the necessity of determining what constitutes valid or careless individual EMA responses to ensure validity and replicability of findings. Currently, EMA studies solely consider the response rate of a participant for exclusion. Yet, other features of an assessment can help to determine whether a response is careless or implausible. Here, we examined over 18,000 EMA text message responses of individual affect items to derive a data-driven model of what constitutes a "careless response." Results from this study indicate that an overly fast time to complete items (#1 s), an overly narrow within assessment response variance (SD # 5), and the percentage of items that fall at the mode ($60%) are independent and reliable indicators of a careless response. Excluding careless responses such as these remove implausible positive correlations among psychometric antonyms (e.g., relaxed and anxious). Further, by identifying and removing careless responses, we also identify careless responders, participants who could be removed from group analyses. We use these results to develop and introduce an R package, EMAeval, so EMA researchers may similarly identify careless responses and responders either online during data collection or posthoc, after data collection has completed.

The distribution of daily affect distinguishes internalizing and externalizing spectra and subfactors.

“…The distribution of daily affect assessed via EMA, both of the entire sample and after having removed careless responses, is presented in Figure 2. Kurtosis is the only distributional EMA metric that appeared to shift after removing the 478 (2.93%) plausibly careless responses (Jaso et al, in press; defined as: <1 s time per item, within EMA SD ≤ 5, proportion of modal items ≤ 60%), leaving 15,834 total EMAs. Correlations between EMA metrics are displayed in Table 2…”

“…We performed our analyses using both full EMA data as well as EMA data after having removed plausibly careless responses. We have found (Jaso et al, in press) that an individual EMA may be careless if it is completed too quickly (≤1 s taken per item), if there is too little variance across item responses ( SD ≤ 5 if items are on a 100-point scale), or if a substantial proportion of items are rated at the modal response (≥60% items rated at the mode). While there were few EMAs removed in total (<3% of EMAs), and the distribution of EMA data did not change dramatically, removing careless responses altered some associations with psychiatric spectra and syndromes.…”

“…We have found that individual EMAs may be completed carelessly and are implausible reflections of one’s current affective state (Jaso et al, in press). Determinants of a careless EMA include: (a) an EMA completed in which the participant spent less than an average of 1 s per item; (b) any EMA in which the within-EMA standard deviation is less than 5 (when using a 0–100 scale); and (c) any EMA in which more than 60% of items are given the same modal score.…”

Temporal dynamics of depressive symptomatology: An idiographic time series analysis applying network models to patients with depressive disorders

“…While it is theoretically possible that the depressive symptomatology of this individual changes fast and that no day, week, or month is alike, there are various other, at least equally likely reasons for these results. These include the possibility of fundamental issues with assessment, such as the inappropriateness of our measures or inaccurate responding (Jaso et al, 2021). Irrespective of its root cause, the nature of the results of participant 2 and, to a degree, other participants with small bandwidths, can hamper the usefulness of the present method.…”

Assessing the reliability of single-item momentary affective measurements in experience sampling.

“…These random sources likely add more measurement error to people’s responses, concealing changes in participants’ true emotional experience. Second, because its repeated assessments make ESM quite an intensive data collection method, researchers aim to keep participant burden and careless responding to a minimum (Eisele et al, 2022; Jaso et al, 2021). Contrary to traditional cross-sectional questionnaires, this leads ESM researchers to opt for single-item momentary emotion constructs (Wilhelm & Schoebi, 2007).…”