Applying systems theory to the evaluation of a whole school approach to violence prevention

Abstract

Issue addressed Our Watch led a complex 12‐month evaluation of a whole school approach to Respectful Relationships Education (RRE) implemented in 19 schools. RRE is an emerging field aimed at preventing gender‐based violence. This paper will illustrate how from an implementation science perspective, the evaluation was a critical element in the change process at both a school and policy level. Methods Using several conceptual approaches from systems science, the evaluation sought to examine how the multiple systems layers – student, teacher, school, community and government – interacted and influenced each other. A distinguishing feature of the evaluation included ‘feedback loops’; that is, evaluation data was provided to participants as it became available. Evaluation tools included a combination of standardised surveys (with pre‐ and post‐intervention data provided to schools via individualised reports), reflection tools, regular reflection interviews and summative focus groups. Results Data was shared during implementation with project staff, department staff and schools to support continuous improvement at these multiple systems levels. In complex settings, implementation can vary according to context; and the impact of evaluation processes, tools and findings differed across the schools. Interviews and focus groups conducted at the end of the project illustrated which of these methods were instrumental in motivating change and engaging stakeholders at both a school and departmental level and why. Conclusion The evaluation methods were a critical component of the pilot's approach, helping to shape implementation through data feedback loops and reflective practice for ongoing, responsive and continuous improvement. Future health promotion research on complex interventions needs to examine how the evaluation itself is influencing implementation. So what? The pilot has demonstrated that the evaluation, including feedback loops to inform project activity, were an asset to implementation. This has implications for other health promotion activities, where evaluation tools could be utilised to enhance, rather than simply measure, an intervention. The findings are relevant to a range of health promotion research activities because they demonstrate the importance of meta‐evaluation techniques that seek to understand how the evaluation itself was influencing implementation and outcomes.

The influence of contextual factors on implementation fidelity in a whole school approach to prevention of violence against women

“…Schools were engaged by different elements of the implementation objects. For some, it was primarily the curriculum component, for others, it was the results of the survey administered to every staff member on gender equity in the school and for other schools, the implementation leader together with the audit process was a key means of engagement.…”

“…Another theme to emerge from the study was how the evaluation materials influenced implementation, with different schools utilising these resources in different ways . For some schools with little previous experience or understanding of the topic, it was the all‐staff survey results on gender equity that were a revelation and prompted schools to begin a change process …”

“…Some schools responded well to the professional development provided as part of the RREiS program and others less so. Certain schools responded well to the policy audit process while others were less engaged . Again, a fidelity to function approach would suggest that finding different ways of structuring a reflection process on school policies and culture might work better than mandating the use of a particular audit tool.…”

“…Given that adaptations are recommended to suit different characteristics of program staff, the community, and the school contexts (including stages of readiness for change), having the data to highlight where things are working (or not working) will be critical in ensuring that the best possible outcomes are achieved. Each setting would require an action research type process of integrating implementation and evaluation akin to action research cycles so that they can continually monitor whether their adaptions to the project are working and modify accordingly, which was the approach used in RREiS . It would also be useful to monitor how the program itself influences the local context.…”

“…It uses the education system as a catalyst for generational and cultural change by engaging schools, as both education institution and workplaces to comprehensively address the drivers of gender‐based violence and create a future free from such violence. Respectful Relationships Education is often considered as only what is delivered in the classroom; however, evidence shows that to achieve maximum effectiveness, RRE should be realised through a whole school approach, which holistically addresses the culture, policies, and community partnerships alongside curriculum . The 2015 Respectful Relationships Education in Schools (RREiS) project was based on an earlier pilot program on RRE implemented in 2010 .…”