Rethinking policy-related research: charting a path using qualitative comparative analysis and complexity theory

Abstract

This article argues that conventional quantitative and qualitative research methods have largely failed to provide policy practitioners with the knowledge they need for decision making. These methods often have difficulty handling real world complexity, especially complex causality. This is when the mechanism of change is a combination of conditions that occur in a system such as an organisation or locality. A better approach is to use Qualitative Comparative Analysis (QCA), a hybrid qualitative/quantitative method that enables logical reasoning about actual cases, their conditions, and how outcomes emerge from combinations of these conditions. Taken together these comprise a system, and the method works well with a whole system view, avoiding reductionism to individual behaviours by accounting for determinants that operate at levels beyond individuals. Using logical reduction, QCA identifies causal mechanisms in sub-types of cases differentiated by what matters to whether the outcome happens or not. In contrast to common variable-based methods such as multiple regression, which are divorced from actual case realities, QCA is case-based and rooted in these realities. The use of qualitative descriptors of conditions such as ways of working engages practitioners, while their standardisation enables systematic comparison and a degree of generalisation about 'why' questions that qualitative techniques typically do not achieve. The type of QCA described in the article requires conditions and outcomes to be dichotomised as present or absent, which is helpful to practitioners facing binary decisions about whether to do (a) or (b), or whether or not an outcome has been achieved.

Integrating quasi-experimental and inductive designs in evaluation: A case study of the impact of free bus travel on public health

“…This specification is what aids those in other jurisdictions in evaluating whether these conditions are likely to hold. Further comparisons with other cases (such as conditional concessions in other cities) will enable these claims about transferability to be extended and strengthened, through the use of, for instance, Qualitative Comparative Analysis (Blackman, 2013;Blackman et al, 2013;Ragin, 1999). However, we have shown how comparisons within the case can also be used to aid the causal claims made.…”

“…To assess whether an intervention implemented in one setting is likely to have similar effects elsewhere, decision makers need evidence on the underlying principles by which an intervention works and on the key contextual factors which influence its effects. For these reasons, there have been trenchant critiques of the prioritization of experimental design in general in evidence-based policy, with calls for developments of case-based analyses such as Qualitative Comparative Analysis to strengthen transferability (Blackman, 2013;Blackman et al, 2013;Byrne, 2013;Woolcock, 2013).…”

Participatory systems mapping for complex energy policy evaluation

“…There is a range of different methods which are taken from, or inspired by, complexity science and applied in evaluation. The most well-used include qualitative comparative analysis (Befani, 2013; Blackman, 2013; Blackman et al, 2013; Byrne, 2013), process tracing (Schmitt and Beach, 2015), case study approaches (Woolcock, 2013), approaches to working with stakeholders (Copestake, 2014), agent-based modelling (Morell et al, 2010), and social network analysis (Drew et al, 2011; Durland and Fredericks, 2005).…”

Moving Away from Social Work and Half Way Back Again: New Research on Skills in Probation

“…Qualitative research alone will not find it easy to provide evidence of effectiveness. As a recent discussion of policy-related research pointed out: Qualitative research is often advocated as the best way to capture the complexity of social phenomena, but even rich case studies full of insight about how things happen are very limited in answering ‘why’ questions without the systematic comparison of cases that would enable us to understand causation, which is essential to policy intervention ( Blackman, 2013 , p. 334). …”