A New Realistic Evaluation Analysis Method

Abstract

In attempting to use a realistic evaluation approach to explore the role of Community Parents in early parenting programs in Toronto, a novel technique was developed to analyze the links between contexts (C), mechanisms (M) and outcomes (O) directly from experienced practitioner interviews. Rather than coding the interviews into themes in terms of context, intervention elements (mechanisms) and outcomes separately and which could be assembled into CMO configurations by the analyst, they were coded as linked dyads and triads directly from the practitioner narratives. Out of all of the linked codes entered, there were a maximum of three with the same combination, presenting challenges for typical qualitative data analysis. This article examines a novel technique that was developed in an attempt to expand this method beyond the circumstances described in the realistic evaluation literature to date. The bulk of the article focuses on the linked coding and analysis procedures, the challenges faced, and the original solutions that were developed to analyze the CMO relations and generate the mid-range theories necessary to move to the next stage of a realist evaluation approach. The features that distinguish this linked coding method from other methods (e.g. Qualitative Comparative Analysis), the major benefits and drawbacks, the utility of the approach within evaluation practice, and its application to realist synthesis and research are discussed. Downloaded from and published evidence, the literature typically describes a qualitative interview process with the plausible hypotheses derived logically from the data. Pawson and Tilley describe an interactive interview or dialogue as a method to understand context, mechanism, and outcome relations where the interviewer and the interviewee co-develop these hypotheses (Pawson & Tilley, 1997). If many practitioners are involved, is there another way that these relations can be uncovered using interviews with a group of practitioners?Several methodological problems are posed in trying to do this. One is the need to ask the right questions to collect information about the contexts, and the program activities and characteristics that could become mechanisms. A second is how to recognize the CMO connections that are made implicitly or explicitly in the data. A third major problem is how to analyze connected strings of data in a way that is different from grouping codes under common themes. This article will examine a novel technique that was developed in an attempt to expand this method beyond the circumstances described in the realistic evaluation literature to date. We begin by examining the realist evaluation literature related to coding, and then describe the case and data collection methods. The bulk of the article focuses on the analysis methods, the challenges we faced, and the solutions that were developed. We end by describing the strengths and limitations of the linked coding approach and suggest implications for evaluation practice and future research.

A realist evaluation of a physical activity participation intervention for children and youth with disabilities: what works, for whom, in what circumstances, and how?

“…Data for realist evaluation is typically collected using qualitative approaches [23], and in this study an ethnographic approach was adopted. Ethnographic methods have demonstrated utility in describing the process of change during an intervention, and how and why an intervention ‘works’ [21, 24, 25].…”

The processes of implementing and sustaining an intensive volunteer one-to-one support (doula) service for disadvantaged pregnant women

“…These emergent themes contributed to the development of theories about the Mechanisms that might result in particular Outcomes in particular Contexts (Pawson and Manzano-Santaella, 2012), using Jackson and Kolla's technique of coding data extracts for each element of the CMO configuration (Jackson and Kolla, 2012). In the presentation of Findings, the letters (C), (M), (O) are used to highlight Contexts, Mechanisms and Outcomes.…”

Employee Transition Throughout an Appreciative Inquiry Intervention

“…They then tested the CCMOC by searching for positive and negative cases. Jackson and Kolla (2012) highlight a difficulty with this approach as being overly reliant on the analysts for the development of CCMOCs, particularly where there is little literature to develop an adequate programme theory. They recommend an approach in which contextmechanism-outcome links are identified directly from the data.…”

“…This enabled the researcher to discuss the CCMOCs-initially developed by the researcher from the pre-AI and during-AI data-in detail with the participants during the interview phase. This is a methodological contribution to the field of realist evaluation which helped to overcome Jackson and Kolla's (2012) concern that approaches to data analysis in realist evaluations either rely too much on the analyst to develop the contextmechanism-outcome links or are overly time consuming. …”