Using systems methods to elicit complex program theories

Abstract

Contextual interventions pose unique challenges for implementation studies and evaluation research. By contextual, we mean programs that are tightly integrated with the social systems they are seeking to influence or transform. This integration challenges foundational assumptions of intervention science and necessitates the development of innovative approaches to monitoring and evaluation. The W3 (‘What Works and Why’) project piloted the use of complex systems theory and methods (CSTM) to develop program theories for peer and community‐led (PCL) programs in HIV and viral hepatitis prevention and health promotion. As a class, these programs have struggled to articulate and theorize their approach to justify their funding and to build an evidence base. CSTM allowed us to articulate detailed program theories based on ‘mental models’ held by program staff and stakeholders. This chapter describes our reflection and adaptation experience as we piloted the CSTM approach in─and as─social research.

Tackling structural stigma: a systems perspective

“…To validate the maps, the workshops explored system dynamics by participants selecting an issue from their practice and a starting place on the map, and then following the pathways laid out by arrows and items, identifying and discussing the implications for that issue. The workshops tested hypotheticals, asking “if this element suddenly stopped working, what might happen elsewhere in the system?” Participants drew on their peer work experience and the maps were refined where required to reflect the participant's experience [18]. These implications identified through the system mapping formed part of the system descriptions.…”

“…Drawing on the realist evaluation work of Pawson and Tilley [24], we analysed the full set of complex system maps (for an example, see Figure 1) to identify the key underlying functions that occurred across all the system maps and which enable peer‐led responses to be effective and sustainable in continually changing community and policy environments (W3 framework, Figure 2). The methodology of stage 1 has been described in detail elsewhere [18, 25, 26].…”

“…Consistent with the system mapping methodology [17, 18] as a form of research co‐production, the project followed a dynamic and iterative process in constant conversation between peer practitioners and the research team to develop “system maps.” The system maps and their accompanying text descriptions are the qualitative data analysed in this paper.…”

“…The workshops tested hypotheticals, asking "if this element suddenly stopped working, what might happen elsewhere in the system?" Participants drew on their peer work experience and the maps were refined where required to reflect the participant's experience [18]. These implications identified through the system mapping formed part of the system descriptions.…”

“…Organizations in the W3 collaboration and individual participants from each organization provided signed informed consent to participate in workshops at each stage. Consistent with the system mapping methodology [17,18] as a form of research co-production, the project followed a dynamic and iterative process in constant conversation between peer practitioners and the research team to develop "system maps." The system maps and their accompanying text descriptions are the qualitative data analysed in this paper.…”