Supplemental Material for A Power Analysis for Fidelity Measurement Sample Size Determination

Abstract

The importance and complexity of assessing fidelity has been emphasized in recent years, with increasingly sophisticated definitions of fidelity (Dane & Schneider, 1998; Durlak & Dupre, 2008; O'Donnell, 2008) and recommended procedures for developing fidelity instruments and collecting fidelity data (Nelson, Cordray, Hulleman, Darrow, & Sommer, 2012). Researchers agree that in order to better understand an intent-to-treat study, measurement should be spread across the entire study period (Gersten, Baker, & Lloyd, 2000; Nelson, et al. 2012); however, little guidance has been provided about how to determine the number of observations needed to precisely measure fidelity (Smith, Daunic, & Taylor, 2007). With limited resources for research, this is an important question, particularly for interventions that last a considerable length of time. Increasingly, these data are being used to enhance the analysis of outcomes. This paper proposes a method for determining a reasonable sample size for fidelity data collection, in the case that fidelity assessment requires observation and coding of instructional sessions either live or by videotape. The proposed methodology is based on consideration of the power of tests of the treatment effect of outcome itself, as well as fidelity's contribution to the variability of outcomes. Software for the sample size calculation is provided.