Why Does Construct Validity Matter in Measuring Implementation Fidelity? A Methodological Case Study

Abstract

In reading intervention research, implementation fidelity is assumed to be positively related to student outcomes, but the methods used to measure fidelity are often treated as an afterthought. Fidelity has been conceptualized and measured in many different ways, suggesting a lack of construct validity. One aspect of construct validity is the fidelity index of a measure. This methodological case study examined how different decisions in fidelity indices influence relative rank ordering of individuals on the construct of interest and influence our perception of the relation between the construct and intervention outcomes. Data for this study came from a large State-funded project to implement multi-tiered systems of support for early reading instruction. Analyses were conducted to determine whether the different fidelity indices are stable in relative rank ordering participants and if fidelity indices of dosage and adherence data influence researcher decisions on model building within a multilevel modeling framework. Results indicated that the fidelity indices resulted in different relations to outcomes with the most commonly used fidelity indices for both dosage and adherence being the worst performing. The choice of index to use should receive considerable thought during the design phase of an intervention study.

From research to daily clinical practice: implementation of orthogeriatric co-management in the trauma ward

“…The third and fourth core component were the development of an individual care plan and systematic follow-up, respectively. Adherence to the core components in at least 80% of the patients was needed to consider the program as feasible ( 34 ). We also evaluated the adherence to the care processes as described in the research protocol ( 21 ).…”

Commentary on “Overcoming implementation challenges through using a train-the-trainer approach to teach numeracy in a special school setting”

“…The concept of implementation fidelity sounds so simple; did the intervention occur as intended? Yet, like many aspects of experimentation this simple construct has been difficult to operationally define and measure (Van Dijk et al, 2022). One theme that emerges from the definitions provided for implementation fidelity is that it involves examining whether the intervention was implemented as intended, and the evaluation must be multidimensional to capture all aspects of implementation.…”

Re-Examining the Reliability and Validity of the SWPBIS Tiered Fidelity Inventory

“…Practitioners can use fidelity data to determine how well a program is implemented and to target professional development to increase fidelity. For researchers, fidelity helps to establish internal validity (Shadish et al, 2002) and identify mechanisms of multi-component interventions critical to achieving positive outcomes (van Dijk et al, 2022). When practices are implemented with poor fidelity, judgments of their effectiveness may be inaccurate (Sanetti et al, 2021).…”