Psychometric Evaluation of the TWente Engagement with Ehealth Technologies Scale (TWEETS): Evaluation Study

Abstract

Background Engagement emerges as a predictor for the effectiveness of digital health interventions. However, a shared understanding of engagement is missing. Therefore, a new scale has been developed that proposes a clear definition and creates a tool to measure it. The TWente Engagement with Ehealth Technologies Scale (TWEETS) is based on a systematic review and interviews with engaged health app users. It defines engagement as a combination of behavior, cognition, and affect. Objective This paper aims to evaluate the psychometric properties of the TWEETS. In addition, a comparison is made with the experiential part of the Digital Behavior Change Intervention Engagement Scale (DBCI-ES-Ex), a scale that showed some issues in previous psychometric analyses. Methods In this study, 288 participants were asked to use any step counter app on their smartphones for 2 weeks. They completed online questionnaires at 4 time points: T0=baseline, T1=after 1 day, T2=1 week, and T3=2 weeks. At T0, demographics and personality (conscientiousness and intellect/imagination) were assessed; at T1-T3, engagement, involvement, enjoyment, subjective usage, and perceived behavior change were included as measures that are theoretically related to our definition of engagement. Analyses focused on internal consistency, reliability, and the convergent, divergent, and predictive validity of both engagement scales. Convergent validity was assessed by correlating the engagement scales with involvement, enjoyment, and subjective usage; divergent validity was assessed by correlating the engagement scales with personality; and predictive validity was assessed by regression analyses using engagement to predict perceived behavior change at later time points. Results The Cronbach alpha values of the TWEETS were .86, .86, and .87 on T1, T2, and T3, respectively. Exploratory factor analyses indicated that a 1-factor structure best fits the data. The TWEETS is moderately to strongly correlated with involvement and enjoyment (theoretically related to cognitive and affective engagement, respectively; P<.001). Correlations between the TWEETS and frequency of use were nonsignificant or small, and differences between adherers and nonadherers on the TWEETS were significant (P<.001). Correlations between personality and the TWEETS were nonsignificant. The TWEETS at T1 was predictive of perceived behavior change at T3, with an explained variance of 16%. The psychometric properties of the TWEETS and the DBCI-ES-Ex seemed comparable in some aspects (eg, internal consistency), and in other aspects, the TWEETS seemed somewhat superior (divergent and predictive validity). Conclusions The TWEETS performs quite well as an engagement measure with high internal consistency, reasonable test-retest reliability and convergent validity, good divergent validity, and reasonable predictive validity. As the psychometric quality of a scale is a reflection of how closely a scale matches the conceptualization of a concept, this paper is also an attempt to conceptualize and define engagement as a unique concept, providing a first step toward an acceptable standard of defining and measuring engagement.

User Experience, Engagement, and Popularity in Mental Health Apps: Secondary Analysis of App Analytics and Expert App Reviews

“…A scoping review of concepts and components of engagement used in the digital health literature emphasized that engagement is a multifaceted concept with behavioral, cognitive, and affective components [ 7 ]. Although self-report engagement measures aim to capture some of these components [ 8 ], in this paper we focus on the behavioral component using analytic data, which tracks users’ actual usage of apps (eg, number of downloads or average time per use). However, analytic data can also be used to determine different conceptualizations of engagement, and we consider some of these approaches below.…”

Improving the Development and Implementation of Audit and Feedback Systems to Support Health Care Workers in Limiting Antimicrobial Resistance in the Hospital: Scoping Review

“…However, these constructs are important for how an AF system is received and used in practice [ 74 ]. The lack of design aspects and considerations of engagement might reflect the neglect of eHealth system characteristics (such as design) and engagement as active influencers for eHealth effectivity [ 75 ]. Second, comparison, goal setting, and action planning were hardly described in the included studies.…”

Targeting subjective engagement in experimental therapeutics for digital mental health interventions

“…Consequently, results substantiate the need for clinical research to move beyond only targeting and assessing objective use of DMHIs to also target the user experience ( Yardley et al, 2016 ). New measures have been created to capture multiple components of engagement in digital interventions ( Kelders and Kip, 2019 ; Kelders et al, 2020a ; Perski et al, 2020 ), which can improve tests of the relation between engagement and outcomes.…”