Using logic model mapping to evaluate program fidelity

Holliday, Lisa R.

doi:10.1016/j.stueduc.2014.04.001

Cited by 11 publications

(15 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This mixed methods evaluation examined the intervention delivery fidelity of a consumer-focused web application with data integration from the primary health care EHR and optional health message receipt. No single or uniform measure defines or quantifies fidelity [ 20 , 33 ], but core evaluation metrics of reach, content fidelity, and dose delivered [ 19 , 30 ] overall were fulfilled as intended. Adaptations to routine implementation, known as structural adaptations [ 34 ], were made to overcome individual barriers to intervention reach and receipt.…”

Section: Discussionmentioning

confidence: 99%

“…Qualitative data revealed important influencing factors on intervention delivery and receipt, a noted advantage of mixed methods inquiry in process evaluations [ 13 ], and a strength of this study. The original program logic model was used as not only a representation of the causal assumptions within the intervention [ 13 ] but also as a scaffold on which to show where the identified factors impacted intervention delivery and uptake [ 20 ].…”

Section: Discussionmentioning

confidence: 99%

“…Participant factors affecting intervention delivery and exposure were gathered from textual data within the email and telephone records. These 2 fidelity measures closely relate to the 2 stages of a program’s logic model: the intervention inputs and the intervention uptake [ 20 ].…”

Section: Methodsmentioning

confidence: 99%

“…Both dimensions can influence future program adaptation and scale-up [ 19 ]. Other proposed benefits of measuring or evaluating fidelity include ensuring internal program validity [ 20 ] and identifying which intervention elements affected participant responsiveness or reaction [ 21 ]. It is further recommended that these analyses precede trial outcome assessments to avoid bias in their interpretation [ 13 , 22 ].…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

An Internet-Based Intervention for Cardiovascular Disease Management Integrated With Primary Care Electronic Health Records: Mixed Methods Evaluation of Implementation Fidelity and User Engagement

Coorey¹,

Peiris²,

Scaria³

et al. 2021

J Med Internet Res

View full text Add to dashboard Cite

Background Growing evidence supports the benefits of eHealth interventions to increase patient engagement and improve outcomes for a range of conditions. However, ineffective program delivery and usage attrition limit exposure to these interventions and may reduce their effectiveness. Objective This study aims to evaluate the delivery fidelity of an eHealth intervention, describe use patterns, compare outcomes between low and high users, and identify mediating factors on intervention delivery and receipt. Methods This is a mixed methods study of an internet-based intervention being evaluated for effectiveness in a randomized controlled trial (RCT). The intervention comprised medication and cardiovascular disease (CVD) risk data uploaded from the primary care electronic health record (EHR); interactive, personalized CVD risk score estimation; goal setting and self-monitoring; an interactive social forum; and optional receipt of heart health messages. Fidelity was assessed over 12 months. Trial outcomes were compared between low and high users. Data sources included program delivery records, web log data, trial data, and thematic analysis of communication records. Results Most participants in the intervention group (451/486, 93%) had an initial training session conducted by telephone (413/447, 92.4% of participants trained), with a mean duration of 44 minutes (range 10-90 minutes). Staff conducted 98.45% (1776/1804) of the expected follow-ups, mostly by telephone or email. Of the 451 participants who commenced log-ins, 46.8% (211) were categorized as low users (defined as at least one log-in in 3 or fewer months of follow-up), 40.4% (182) were categorized as high users (at least one log-in in more than 3 months of follow-up), and 12.8% (58) were nonadopters (no log-ins after their training session). The mean log-in frequency was 3-4 per month in ongoing users. There was no significant difference between the groups in the primary trial outcome of adherence to guideline-recommended medications (P=.44). In unadjusted analyses, high users had significantly greater eHealth literacy scores (P=.003) and were more likely to meet recommended weekly targets for fruit (P=.03) and fish (P=.004) servings; however, the adjusted findings were not significant. Interactive screen use was highest for goal tracking and lowest for the chat forum. Screens with EHR-derived data held only an early interest for most users. Fidelity measures (reach, content, dose delivered, and dose received) were influenced by the facilitation strategies used by staff, invisible qualities of staff-participant communication, and participants’ responsiveness to intervention attributes. Conclusions A multifeature internet-based intervention was delivered with high fidelity to the RCT protocol and was regularly used by 40.4% (182/451) of users over 12 months. Higher log-in frequency as an indicator of greater intervention exposure was not associated with statistically significant improvements in eHealth literacy scores, lifestyle changes, or clinical outcomes. Attributes of the intervention and individualized support influenced initial and ongoing use.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

An Internet-Based Intervention for Cardiovascular Disease Management Integrated With Primary Care Electronic Health Records: Mixed Methods Evaluation of Implementation Fidelity and User Engagement

Coorey¹,

Peiris²,

Scaria³

et al. 2021

J Med Internet Res

View full text Add to dashboard Cite

show abstract

“…Program logic models are routinely applied across diverse settings and disciplines, including to operationalise multi-site school interventions (Holliday 2014), research programming (O'Keefe and Head 2011), public health promotion (Crosby and Noar 2011), conservation planning (Margoluis et al 2009), forensic treatment (Stinchcomb 2001) and organisational development and coaching (Oosthuizen and Louw 2013) as just some examples. Logic models offer a method to support agencies or work teams to develop a shared understanding of the program model, including its delivery and underpinning philosophies (English and Kaleveld 2003;McLaughlin and Jordan 2004).…”

Section: Program Logic Modellingmentioning

confidence: 99%

Program Logic Modelling and Complex Positive Psychology Intervention Design and Implementation: The ‘Resilient Futures’ Case Example

Raymond¹,

Iasiello

Kelly

et al. 2019

Int J Appl Posit Psychol

View full text Add to dashboard Cite

Positive psychology interventions (PPIs) and programs differ markedly in implementation quality. Lower quality implementation is associated with interventions that include multiple components (or PPIs), are delivered across multiple layers (individual, workgroup/ classroom and organisation/school) or agency sites, and include cohorts with heterogeneous or complex needs (e.g., trauma). This paper collectively refers to these interventions as 'complex PPI's' or 'complex programs'. Drawing upon the implementation science literature, we argue that logic modelling represents a method to guide the design and implementation of 'complex' interventions. We describe a growth-focused logic model and implementation methodology (titled intentional practice) that operationalises positive psychology outcomes and processes to support program developers to achieve a balance between fidelity and adaptation. The Resilient Futures program, developed by the South Australian Health and Medical Research Institute (SAHMRI) Wellbeing and Resilience Centre, is described as a case example of the approach. This program represents a large scale and multi-site implementation of a resilience and wellbeing skill-building program for 850 young people from disadvantaged backgrounds with heterogeneous needs. The case study describes how logic modelling and intentional practice consolidated early theory building work to operationalise the intervention at the program level, but also to support multi-site nuancing and translation of the program to individual sites and agency/youth needs. The article argues that logic modelling offers a flexible and evidence-based method for program designers, facilitators and researchers to design and implement complex PPI's or programs. Key considerations for utilising logic models are offered.

show abstract

Using logic model and visualization to conduct portfolio evaluation

Shen

Jones

et al. 2019

Evaluation and Program Planning

View full text Add to dashboard Cite

Using logic model mapping to evaluate program fidelity

Cited by 11 publications

References 27 publications

An Internet-Based Intervention for Cardiovascular Disease Management Integrated With Primary Care Electronic Health Records: Mixed Methods Evaluation of Implementation Fidelity and User Engagement

An Internet-Based Intervention for Cardiovascular Disease Management Integrated With Primary Care Electronic Health Records: Mixed Methods Evaluation of Implementation Fidelity and User Engagement

Program Logic Modelling and Complex Positive Psychology Intervention Design and Implementation: The ‘Resilient Futures’ Case Example

Using logic model and visualization to conduct portfolio evaluation

Contact Info

Product

Resources

About