Quality of implementation: developing measures crucial to understanding the diffusion of preventive interventions

Abstract

As prevention programs become disseminated, the most serious threat to effectiveness is maintaining the quality of implementation intended by the developers. This paper proposes a methodology for measuring quality of implementation in school settings and presents data from a pilot study designed to test several of the proposed components. These methods included assessments of adherence, quality of process, the positive or negative valence of adaptations, teachers' attitudes and teachers' understanding of program content. This study was conducted with 11 teachers who had varying degrees of experience who taught Life Skills Training. Observation and interview data were collected during visits to schools. Results suggest that quality of implementation can be measured through observation and interview. Teachers varied in adherence and quality of program delivery. All teachers made adaptations to the program. Experienced teachers were more likely to adhere to the curriculum, deliver it in a way that was more interactive and engaging to students, communicate the goals and objectives better, and make positive adaptations. The field can use these findings as the basis for exploring strategies for measuring and improving quality of implementation.

Evaluación de proceso del programa escolar de prevención del consumo de cánnabis “xkpts.com” en adolescentes de Barcelona en 2006

“…Nuestros resultados muestran que a mayor cumplimiento de la implementación mayor es la satisfacción del profesorado con la intervención. Sin embargo, tan sólo en dos de cada cinco aulas se realizó una intervención cualificada, dato que concuerda con la evidencia científica que muestra la dificultad de conseguir una implementación adecuada en las intervenciones preventivas escolares 10,[14][15][16] . En el caso del criterio cuantitativo de implementación, que evalúa la intervención en función del número de actividades realizadas, la intervención fue correcta en tres de cada cinco aulas.…”

“…No obstante, consideramos que una de las fortalezas del estudio es el tamaño de muestra, con la participación de 39 escuelas, muestra muy superior a la de otros estudios 11,14,16,17,39,40 .…”

“…En los últimos años un elevado número de estudios han señalado la importancia de evaluar la fidelidad de la implementación de las inter venciones preventivas escolares [13][14][15][16][17][18] . Su evaluación nos puede conducir a un mejor entendimiento del éxito o fracaso de un programa ya que la obtención de los resultados esperados está estrechamente vinculada a la intensidad y cualidad de la implementación 15,19 .…”

A Randomized Controlled Trial Evaluation of an After-School Prosocial Behavior Program in an Area of Socioeconomic Disadvantage

“…It is widely accepted that the effectiveness of SEL programs depends on the quality of implementation Derzon, Sale, Springer, & Brounstein, 2005;Dusenbury, Brannigan, Hansen, Walsh, & Falco, 2005;Farrell, Meyer, Kung, & Sullivan, 2001;Gottfredson & Gottfredson, 2002). Furthermore, meta-analyses have provided evidence that the outcomes of programs linked to personal and social learning are limited by implementation quality (Durlak & DuPre, 2008;Durlak et al, 2010Durlak et al, , 2011.…”

Interim evaluation of the Tier 1 Program of Project P.A.T.H.S.: continuation of evidence

“…These are important elements of program evaluation that help researchers monitor program adherence (a key component of program success), create infrastructure that supports the project, evaluate how effectively that process functions, and assess changes in skills, attitudes and knowledge of the participants and the program implementers (8,9) . As Gomby and Larson (10) described, " process evaluation focuses on what services were provided to whom and how.…”