A novel method for assessing design fidelity in web-based behavioral interventions.

Lambert, Jeffrey; Elliott, Lewis R.; Taylor, Adrian; Farrand, Paul; Haase, Anne M.; Greaves, Colin

doi:10.1037/hea0001046

Cited by 12 publications

(7 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This supports the notion that unclear theoretical underpinnings might result in a drift in programme fidelity. 7,28,29 Although many studies evaluating programme designs have focused on coding BCTs, 25,30 we additionally assessed fidelity to service parameters using the TIDieR 21 framework. As we found several important service parameters to not demonstrate fidelity to the service specification, this illustrates other important components of an intervention's protocol 11 outside of BCTs where fidelity might be diluted during the design phase, and not captured using the BCTTv1 8 in isolation.…”

Section: Discussionmentioning

confidence: 99%

Does the design of the NHS Low‐Calorie Diet Programme have fidelity to the programme specification? A documentary review of service parameters and behaviour change content in a type 2 diabetes intervention

et al. 2022

View full text Add to dashboard Cite

Background: NHS England commissioned four independent service providers to pilot low-calorie diet programmes to drive weight loss, improve glycaemia and potentially achieve remission of Type 2 Diabetes across 10 localities. Intervention fidelity might contribute to programme success. Previous research has illustrated a drift in fidelity in the design and delivery of other national diabetes programmes. Aims:(1) To describe and compare the programme designs across the four service providers; (2) To assess the fidelity of programme designs to the NHS England service specification. Methods:The NHS England service specification documents and each provider's programme design documents were double-coded for key intervention content using the Template for Intervention Description and Replication Framework and the Behaviour Change Technique (BCT) Taxonomy. Results:The four providers demonstrated fidelity to most but not all of the service parameters stipulated in the NHS England service specification. Providers included between 74% and 87% of the 23 BCTs identified in the NHS specification.Twelve of these BCTs were included by all four providers; two BCTs were consistently absent. An additional seven to 24 BCTs were included across providers. Conclusions:A loss of fidelity for some service parameters and BCTs was identified across the provider's designs; this may have important consequences for programme delivery and thus programme outcomes. Furthermore, there was a large degree of variation between providers in the presence and dosage of additional BCTs. How these findings relate to the fidelity of programme delivery and variation in programme outcomes and experiences across providers will be examined.

show abstract

Section: Discussionmentioning

confidence: 99%

Does the design of the NHS Low‐Calorie Diet Programme have fidelity to the programme specification? A documentary review of service parameters and behaviour change content in a type 2 diabetes intervention

et al. 2022

View full text Add to dashboard Cite

show abstract

“…To date, research has primarily focused on patient engagement with digital health interventions (eg, via methods such as system usage data, qualitative measures, and self-report questionnaires) [ 28 ] and adherence to digital interventions (ie, the extent to which patients keep using the digital intervention in the desired way) [ 29 ]. Some studies have summarized the development process of digital interventions (eg, describing the theory and existing evidence to inform the development of a digital self-management intervention for people with T2DM) [ 30 ] and compared web-based intervention content against the intervention description [ 31 ]. However, there is a lack of literature examining the extent to which the planned intervention content of digital health interventions adheres to the evidence base for what works in changing health behaviors.…”

Section: Introductionmentioning

confidence: 99%

Fidelity to Program Specification of the National Health Service Digital Diabetes Prevention Program Behavior Change Technique Content and Underpinning Theory: Document Analysis

Hawkes¹,

Miles²,

French³

2022

J Med Internet Res

View full text Add to dashboard Cite

Background The National Health Service (NHS) Diabetes Prevention Program is a behavior change intervention for adults in England who are identified as being at high risk of developing type 2 diabetes. The face-to-face service was launched in 2016, followed by a digital service (NHS Digital Diabetes Prevention Program [NHS-DDPP]) in 2019. A total of 4 service providers were commissioned to deliver the NHS-DDPP and were required to deliver the digital service in line with a program specification detailing the key intervention content. The fidelity of the behavior change content in the digital service (ie, the extent to which the program is delivered as intended) is currently unknown. Digital interventions may allow higher fidelity as staff do not have to be trained to deliver all intervention content. Assessing fidelity of the intervention design is particularly important to establish the planned behavior change content in the NHS-DDPP and the extent to which this adheres to the program specification. This is the first known independent assessment of design fidelity in a large-scale digital behavior change intervention. Objective This study aims to assess the fidelity of the behavior change content in each of the 4 NHS-DDPP providers’ intervention designs to the full program specification. Methods We conducted a document review of each provider’s NHS-DDPP intervention design, along with interviews with program developers employed by the 4 digital providers (n=6). Providers’ intervention design documents and interview transcripts were coded for behavior change techniques (BCTs; ie, the active ingredients of the intervention) using the Behavior Change Technique Taxonomy version 1 and underpinning theory using the Theory Coding Scheme framework. The BCTs identified in each digital provider’s intervention design were compared with the 19 BCTs included in the program specification. Results Of the 19 BCTs specified in the program specification, the 4 providers planned to deliver 16 (84%), 17 (89%), 16 (84%), and 16 (84%) BCTs, respectively. An additional 41 unspecified BCTs were included in at least one of the 4 digital providers’ intervention designs. By contrast, inconsistent use of the underpinning theory was apparent across providers, and none of the providers had produced a logic model to explain how their programs were expected to work. All providers linked some of their planned BCTs to theoretical constructs; however, justification for the inclusion of other BCTs was not described. Conclusions The fidelity of BCT content in the NHS-DDPP was higher than that previously documented for the face-to-face service. Thus, if service users engage with the NHS-DDPP, this should increase the effectiveness of the program. However, given that a clear theoretical underpinning supports the translation of BCTs in intervention designs to intervention delivery, the absence of a logic model describing the constructs to be targeted by specific BCTs is potentially problematic.

show abstract

“…While engagement following download was poor, log data identified that engaged users had moderate to good levels of interaction and fidelity [ 51 ], with the CBT model alongside recording and categorizing worries. Fidelity and interaction with Problem Solving were also good, potentially arising from support provided through the CA.…”

Section: Discussionmentioning

confidence: 99%

Use and Engagement With Low-Intensity Cognitive Behavioral Therapy Techniques Used Within an App to Support Worry Management: Content Analysis of Log Data

Farrand,

Raue,

Ward

et al. 2024

JMIR Mhealth Uhealth

Self Cite

View full text Add to dashboard Cite

Background Low-intensity cognitive behavioral therapy (LICBT) has been implemented by the Improving Access to Psychological Therapies services across England to manage excessive worry associated with generalized anxiety disorder and support emotional well-being. However, barriers to access limit scalability. A solution has been to incorporate LICBT techniques derived from an evidence-based protocol within the Iona Mind Well-being app for Worry management (IMWW) with support provided through an algorithmically driven conversational agent. Objective This study aims to examine engagement with a mobile phone app to support worry management with specific attention directed toward interaction with specific LICBT techniques and examine the potential to reduce symptoms of anxiety. Methods Log data were examined with respect to a sample of “engaged” users who had completed at least 1 lesson related to the Worry Time and Problem Solving in-app modules that represented the “minimum dose.” Paired sample 2-tailed t tests were undertaken to examine the potential for IMWW to reduce worry and anxiety, with multivariate linear regressions examining the extent to which completion of each of the techniques led to reductions in worry and anxiety. Results There was good engagement with the range of specific LICBT techniques included within IMWW. The vast majority of engaged users were able to interact with the cognitive behavioral therapy model and successfully record types of worry. When working through Problem Solving, the conversational agent was successfully used to support the user with lower levels of engagement. Several users engaged with Worry Time outside of the app. Forgetting to use the app was the most common reason for lack of engagement, with features of the app such as completion of routine outcome measures and weekly reflections having lower levels of engagement. Despite difficulties in the collection of end point data, there was a significant reduction in severity for both anxiety (t53=5.5; P<.001; 95% CI 2.4-5.2) and low mood (t53=2.3; P=.03; 95% CI 0.2-3.3). A statistically significant linear model was also fitted to the Generalized Anxiety Disorder–7 (F2,51=6.73; P<.001), while the model predicting changes in the Patient Health Questionnaire–8 did not reach significance (F2,51=2.33; P=.11). This indicates that the reduction in these measures was affected by in-app engagement with Worry Time and Problem Solving. Conclusions Engaged users were able to successfully interact with the LICBT-specific techniques informed by an evidence-based protocol although there were lower completion rates of routine outcome measures and weekly reflections. Successful interaction with the specific techniques potentially contributes to promising data, indicating that IMWW may be effective in the management of excessive worry. A relationship between dose and improvement justifies the use of log data to inform future developments. However, attention needs to be directed toward enhancing interaction with wider features of the app given that larger improvements were associated with greater engagement.

show abstract

A novel method for assessing design fidelity in web-based behavioral interventions.

Cited by 12 publications

References 39 publications

Does the design of the NHS Low‐Calorie Diet Programme have fidelity to the programme specification? A documentary review of service parameters and behaviour change content in a type 2 diabetes intervention

Does the design of the NHS Low‐Calorie Diet Programme have fidelity to the programme specification? A documentary review of service parameters and behaviour change content in a type 2 diabetes intervention

Fidelity to Program Specification of the National Health Service Digital Diabetes Prevention Program Behavior Change Technique Content and Underpinning Theory: Document Analysis

Use and Engagement With Low-Intensity Cognitive Behavioral Therapy Techniques Used Within an App to Support Worry Management: Content Analysis of Log Data

Contact Info

Product

Resources

About