Critical reflections on realist review: insights from customizing the methodology to the needs of participatory research assessment

Abstract

Realist review has increased in popularity as a methodology for complex intervention assessment. Our experience suggests that the process of designing a realist review requires its customization to areas under investigation. To elaborate on this idea, we first describe the logic underpinning realist review and then present critical reflections on our application experience, organized in seven areas. These are the following: (1) the challenge of identifying middle range theory; (2) addressing heterogeneity and lack of conceptual clarity; (3) the challenge of appraising the quality of complex evidence; (4) the relevance of capturing unintended outcomes; (5) understanding the process of context, mechanism, and outcome (CMO) configuring; (6) incorporating middle-range theory in the CMO configuration process; and (7) using middle range theory to advance the conceptualization of outcomes - both visible and seemingly 'hidden'. One conclusion from our experience is that the degree of heterogeneity of the evidence base will determine whether theory can drive the development of review protocols from the outset, or will follow only after an intense period of data immersion. We hope that presenting a critical reflection on customizing realist review will convey how the methodology can be tailored to the often complex and idiosyncratic features of health research, leading to innovative evidence syntheses.

ReseArch with Patient and Public invOlvement: a RealisT evaluation – the RAPPORT study

“…99 Several factors contribute to the weak evidence base for PPI. First, previous evaluations have failed to use methodologies that take into account its complexity, 100 arising from PPI being an interconnected, multifaceted social process, making it difficult to pinpoint specific impacts and contributions.…”

“…19 Second, there is little systematic appraisal and reporting of PPI impact and outcomes, 101 further constrained by the conventions of academic publishing. 21 Nevertheless, there is a limited number of evidence reviews, 19,29,99,101,102 here presented chronologically to illustrate the developing evidence base. The systematic review by Oliver et al 102 was the first to examine the evidence base on the impact of PPI on identifying and prioritising possible topics for research and development.…”

Outreach programmes for health improvement of Traveller Communities: a synthesis of evidence

“…This impacted on our ability to debate issues around the generative causation of certain mechanisms, or indeed whether or not constructs should be classified as, for example, contexts or mechanisms. These are acknowledged difficulties in realist syntheses, 216 which were compounded by our small team size. The correlate to this, however, was our ability to communicate very effectively throughout the project and ensure the timely inclusion of a range of perspectives as the analysis developed.…”

“…In contrast, the literature on outreach is generally undertheorised and lacks process detail and outcomes measurement. Jagosh et al 216 describe how, for their realist review on participatory research, the process of refinement of the research question was a key step in defining study inclusion criteria. In order to increase the homogeneity of our data set, we could have undertaken a similar process of refining the parameters of the review.…”

Explaining how unexploded ordnance clearance enhances livelihoods in the Lao PDR

“…Typically in realist evaluations, programs are presented as a series of implementation 'chains', comprising intervention actions and participant reactions (Jagosh, et al, 2014;Pawson, 2013;Pawson & Tilley, 1997;Weiss, 2000;Wong, et al, 2010).…”