Development and validation of the Overall Fidelity Enactment Scale for Complex Interventions (OFES-CI)

Abstract

BackgroundIn many quality improvement (QI) and other complex interventions, assessing the fidelity with which participants ‘enact’ intervention activities (ie, implement them as intended) is underexplored. Adapting the evaluative approach used in objective structured clinical examinations, we aimed to develop and validate a practical approach to assessing fidelity enactment—the Overall Fidelity Enactment Scale for Complex Interventions (OFES-CI).MethodsWe developed the OFES-CI to evaluate enactment of the SCOPE QI intervention, which teaches nursing home teams to use plan-do-study-act (PDSA) cycles. The OFES-CI was piloted and revised early in SCOPE with good inter-rater reliability, so we proceeded with a single rater. An intraclass correlation coefficient (ICC) was used to assess inter-rater reliability. For 27 SCOPE teams, we used ICC to compare two methods for assessing fidelity enactment: (1) OFES-CI ratings provided by one of five trained experts who observed structured 6 min PDSA progress presentations made at the end of SCOPE, (2) average rating of two coders’ deductive content analysis of qualitative process evaluation data collected during the final 3 months of SCOPE (our gold standard).ResultsUsing Cicchetti’s classification, inter-rater reliability between two coders who derived the gold standard enactment score was ‘excellent’ (ICC=0.93, 95% CI=0.85 to 0.97). Inter-rater reliability between the OFES-CI and the gold standard was good (ICC=0.71, 95% CI=0.46 to 0.86), after removing one team where open-text comments were discrepant with the rating. Rater feedback suggests the OFES-CI has strong face validity and positive implementation qualities (acceptability, easy to use, low training requirements).ConclusionsThe OFES-CI provides a promising novel approach for assessing fidelity enactment in QI and other complex interventions. It demonstrates good reliability against our gold standard assessment approach and addresses the practicality problem in fidelity assessment by virtue of its suitable implementation qualities. Steps for adapting the OFES-CI to other complex interventions are offered.

Towards comprehensive fidelity evaluations: consideration of enactment measures in quality improvement interventions

“…3 There have been calls for the development of highquality measures of engagement (including enactment) that are acceptable and feasible to use, reliable and valid. 3 Therefore, the study by Ginsburg et al 5 extends previous research by considering one way in which it may be possible for researchers to objectively evaluate enactment with high reliability. This is comparable with the gold-standard measure of fidelity of delivery in which multiple researchers reliably rate transcripts of audio/video-recorded intervention sessions.…”

“…These data comprised diary entries from quality advisors, open-ended survey questions completed by research participants and observations conducted by trained members of the research team. 5 The authors demonstrated that the OFES-CI tool was reliable, had face validity and was feasible to implement. 5 Ginsburg et al have filled an important gap in the literature, as a 2017 systematic review found that few previous studies have focused on enactment.…”

“…5 The authors demonstrated that the OFES-CI tool was reliable, had face validity and was feasible to implement. 5 Ginsburg et al have filled an important gap in the literature, as a 2017 systematic review found that few previous studies have focused on enactment. 3 This could be due to a lack of consensus about the role of enactment in fidelity assessments.…”

“…In this issue of BMJ Quality & Safety, Ginsburg et al 5 describe the development and validation of a novel measure of enactment called the 'Overall Fidelity Enactment Scale for Complex Interventions' (OFES-CI). The measure was developed in the context of a quality improvement intervention delivered by healthcare aides in nursing homes to improve resident care, and is intended to be applicable to the full range of quality improvement interventions and other complex interventions.…”

“…Future research will hopefully establish the validity of the new measure for other intervention contexts and participants. 5…”