Practical considerations for evaluating reliability in ambulatory assessment studies.

Abstract

Ambulatory assessment studies have unique challenges related to obtaining adequate reliability relative to other types of research designs. For example, because of time constraints, limitations in the number of items administered at each occasion can negatively affect the reliable assessment of within-person change. Whereas some types of measures, such as objective measures of physical activity, have been the focus of a cumulative body of reliability research, a lack of reporting of reliability data for cognitive and self-report measures in many studies is a significant barrier for researchers who want to choose measures based on prior psychometric research. Several approaches to assessing the reliability of ambulatory assessment data are available, and further research with less widely used reliability metrics (e.g., stability in within-person variability across time) is warranted. Given the time and effort involved in conducting ambulatory assessment studies, proper consideration of reliability is needed so that researchers have sufficient power to take advantage of the large amount of data often available within a single ambulatory assessment study.

Incorporating Ecological Momentary Assessment into multi-method investigations of cognitive aging: Promise and practical considerations

“…In addition, issues of measurement will be paramount when deciding on the type of item and response (e.g., Likert scale, sliding scale) and number of items used to assess a particular construct. There has been little work that has systematically examined the validity and reliability of truncated assessments used in EMA studies (i.e., taking a subset of items from a larger subscale to fit within the time constraints imposed by EMA, although see Calamia, 2019;Wright & Zimmermann, 2019), and none to our knowledge in the context of adult life-span development.…”

Advancing ambulatory biobehavioral technologies beyond “proof of concept”: Introduction to the special section.

“…This special section features four articles. In the first article, Calamia (Calamia, 2019) provides a review of strategies for evaluating the reliability of ambulatory biobehavioral assessments. He focuses on existing solutions (e.g., multilevel modeling, generalizability theory) that have been used within the psychological literature to evaluate reliability with temporally, spatially, or spectrally complex data.…”

Reliability and validity of the Brief Emotional Experience Scale (BEES) for mood monitoring

“…However, such measures used within the mood monitoring literature are rarely subject to more rigorous statistical evaluation via factor analytic techniques, nor is evidence usually provided for convergent validity (i.e., strong associations) with other established mood measures (Calamia, 2019;Dubad et al, 2018;Faurholt-Jepsen et al, 2019;Trull & Ebner-Priemer, 2020). This can act to reduce reader confidence in the reported findings with such measures.…”