Challenges in evaluating implementation and effectiveness in real-world settings: evaluation proposal for school-based health-promoting intervention

Abstract

Summary There are various research designs and approaches to investigate how health-promoting activities are implemented in complex, real-world systems, and to identify potential health effects that might occur following implementation. Although literature describes guidelines to perform and report about implementation research and effect evaluations, no specific guidelines exist on analysing and reporting about the combination of effectiveness data and implementation data collected as part of intervention evaluation in complex and diverse settings. This paper describes the evaluation of primary school-based health-promoting activities in complex systems. Furthermore, an approach for data categorization inspired by Rogers’ Diffusion of Innovations theory is presented that can facilitate structuring the study’s results and relating the degree of implementation to any impact on effectiveness outcomes that might be observed. Researchers interested in using this approach for data categorization have to ensure that the following three conditions are met: (i) data on an intervention’s efficacy in a controlled setting with optimal implementation is available; (ii) key points that define an intervention’s optimal implementation are available and (iii) an evaluation study is performed, collecting both effectiveness data and implementation data in a real-world context. This data categorization approach can be useful to generate more insight into an intervention’s effectiveness under varying circumstances, and optimal support and advice can be provided to stakeholders to achieve maximum impact of population-based health-promoting interventions in complex, real-world systems. However, the proposed approach is a first suggestion and further testing and adaptation is necessary to increase its usefulness. Knowledge and experience sharing among researchers performing comparable research can increase the knowledge base regarding this subject.

Effects of School-Based Health-Promoting Activities on Children’s Health: A Pragmatic Real-World Study

“…In short, these key points entailed (1) active involvement of various stakeholders (e.g., school staff, children, and parents), (2) taking a school-wide approach, (3) all children engaging in at least one hour of PA every day, and (4) all children consuming a daily healthy lunch at school. More details on the categorisation method based on the HPSF key points are described elsewhere [24].…”

“…At the project's start, a strong difference in the intensity of project implementation between the schools was expected, which could subsequently serve as a way to categorise and compare schools. However, limited implementation in all schools restricted the ability to create distinct categories and compare schools using the method previously described elsewhere [24]. Instead, schools were categorised into medium and low implementer groups.…”