Rigour versus the need for evidential diversity

Abstract

This paper defends the need for evidential diversity and the mix of methods that that can in train require. The focus is on causal claims, especially ‘singular’ claims about the effects of causes in a specific setting—either what will happen or what has happened. I do so by offering a template that categorises kinds of evidence that can support these claims. The catalogue is generated by considering what needs to happen for a causal process to carry through from putative cause at the start to the targeted effect at the end. The usual call for mixed methods focusses on a single overall claim and argues that we increase certainty by the use of different methods with compensating strengths and weaknesses. My proposals instead focus on the evidence that supports the great many subsidiary claims that must hold if the overall one is to be true. As is typical for singular causal claims, the mix of methods that will generally be required to collect the kinds of evidence I urge will usually have little claim to the kind of rigour that is now widely demanded in evidencing causal claims, especially those for policy/treatment effectiveness. So I begin with an exploration of what seems to be intended by ‘rigour’ in such discussions, since it is seldom made clear just what makes the favoured methods especially rigorous. I then argue that the emphasis on rigour can be counterproductive. Rigour is often the enemy of evidential diversity, and evidential diversity—lots of it—can make for big improvements in the reliability of singular causal predictions and post hoc evaluations. I illustrate with the paragon of rigour for causal claims, randomised controlled trials (RCTs), rehearsing at some length what they can and cannot do to make it easier to assess the importance of rigour in warranting singular causal claims.

Investigating Tax Compliance with Mixed-Methods Approach: The Effect of Normative Appeals Among the Firms in Latvia

“…On the one hand, we propose a specific way of framing the messages that nudge taxpayers. On the other hand, we agree that part of the answer to tax compliance lies in the system-level changes rather than just identifying the best nudge based on average treatment effect estimates from a trial (Cartwright 2021;Chater and Loewenstein 2022). Therefore, we discuss the structural changes necessary to bring sustainable behavioural change.…”

Evidential Variety and Mixed-Methods Research in Social Science

“…The third source of added value arises when evidence for distinct claims about the same phenomenon together produces a fuller picture of it such that the epistemic whole is better supported by the totality of evidence than the sum of the individual hypotheses (Lloyd 2009, Cartwright 2021). Integration differs from triangulation and security because the evidence need not be independent, nor does it need to be about the same claim.…”

Making and justifying evidence claims: evidence synthesis of impact evaluations and systematic reviews in international development

“…Triangulation can be part of the research design and can also support the validity and interpretation of claims. Cartwright (2021), Kuorikoski and Marchionni (2016) and Claveau (2011) consider the justification of causal inferences and the need for evidential diversity from a mix of methods. Cartwright (2021) focuses on the evidence to support multiple subsidiary claims, and characterise perspectives of rigour in context of claims from RCTs, and their challenges.…”