An Internet-Based Intervention for Cardiovascular Disease Management Integrated With Primary Care Electronic Health Records: Mixed Methods Evaluation of Implementation Fidelity and User Engagement

Abstract

Background Growing evidence supports the benefits of eHealth interventions to increase patient engagement and improve outcomes for a range of conditions. However, ineffective program delivery and usage attrition limit exposure to these interventions and may reduce their effectiveness. Objective This study aims to evaluate the delivery fidelity of an eHealth intervention, describe use patterns, compare outcomes between low and high users, and identify mediating factors on intervention delivery and receipt. Methods This is a mixed methods study of an internet-based intervention being evaluated for effectiveness in a randomized controlled trial (RCT). The intervention comprised medication and cardiovascular disease (CVD) risk data uploaded from the primary care electronic health record (EHR); interactive, personalized CVD risk score estimation; goal setting and self-monitoring; an interactive social forum; and optional receipt of heart health messages. Fidelity was assessed over 12 months. Trial outcomes were compared between low and high users. Data sources included program delivery records, web log data, trial data, and thematic analysis of communication records. Results Most participants in the intervention group (451/486, 93%) had an initial training session conducted by telephone (413/447, 92.4% of participants trained), with a mean duration of 44 minutes (range 10-90 minutes). Staff conducted 98.45% (1776/1804) of the expected follow-ups, mostly by telephone or email. Of the 451 participants who commenced log-ins, 46.8% (211) were categorized as low users (defined as at least one log-in in 3 or fewer months of follow-up), 40.4% (182) were categorized as high users (at least one log-in in more than 3 months of follow-up), and 12.8% (58) were nonadopters (no log-ins after their training session). The mean log-in frequency was 3-4 per month in ongoing users. There was no significant difference between the groups in the primary trial outcome of adherence to guideline-recommended medications (P=.44). In unadjusted analyses, high users had significantly greater eHealth literacy scores (P=.003) and were more likely to meet recommended weekly targets for fruit (P=.03) and fish (P=.004) servings; however, the adjusted findings were not significant. Interactive screen use was highest for goal tracking and lowest for the chat forum. Screens with EHR-derived data held only an early interest for most users. Fidelity measures (reach, content, dose delivered, and dose received) were influenced by the facilitation strategies used by staff, invisible qualities of staff-participant communication, and participants’ responsiveness to intervention attributes. Conclusions A multifeature internet-based intervention was delivered with high fidelity to the RCT protocol and was regularly used by 40.4% (182/451) of users over 12 months. Higher log-in frequency as an indicator of greater intervention exposure was not associated with statistically significant improvements in eHealth literacy scores, lifestyle changes, or clinical outcomes. Attributes of the intervention and individualized support influenced initial and ongoing use.

Improving the management of type 2 diabetes in China using a multifaceted digital health intervention in primary health care: the SMARTDiabetes cluster randomised controlled trial

“…Implementation fidelity compared favourably with the literature with around 50% of intervention participants engaging in the intervention at least monthly throughout the intervention period. 23 FHPs supported one-third of the intervention arm participants. In exploratory analyses to better understand the trial outcomes, use of FHPs and active use of the app was better in rural than urban communities, and significantly associated with improved ABC control rates.…”

Changes in a Digital Type 2 Diabetes Self-management Intervention During National Rollout: Mixed Methods Study of Fidelity

“…This study was conducted at a relatively early phase in the development of Healthy Living, so there may be further changes to the website that would potentially alter these findings. This is a common challenge when assessing the fidelity of digital interventions because of the fast-moving pace of digital technology [ 53 ]. This evaluation was also conducted before the national implementation of primary care referral pathways into Healthy Living, which was delayed because of disruptions caused by the COVID-19 pandemic.…”

Adapting and Evaluating an Artificial Intelligence-Based Chatbot through Patient and Stakeholder Engagement to Provide Information for Different Health Conditions: Master Protocol for an Adaptive Platform Trial (the MARVIN Chatbots) (Preprint)

“…Fidelity is the degree to which the intervention is implemented as intended [ 14 ]. On the basis of the fidelity measures of digital health intervention implementation identified by Coorey et al [ 66 ], we will analyze the following metadata to comprehensively assess the chatbot fidelity: (1) intervention fidelity (the proportion of participants who continue to use the chatbot after a 1-month period), (2) frequency and duration (the monthly frequency with which participants use the chatbot and the average total duration of using the chatbot), (3) messages delivered (the total number of messages that participants interacted with in the chatbot over the course of the study period as well as the average number per participant), and (4) range of messages received (the frequency distribution of different conversational topics triggered by all participants).…”