Notes on the use of randomised controlled trials to evaluate complex interventions: Community treatment orders as an illustrative case

Abstract

Over the past seven decades, randomised controlled trials (RCTs) have revolutionised clinical research and achieved a gold standard status. However, extending their use to evaluate complex interventions is problematic. In this paper we will demonstrate that complex intervention RCTs violate the necessary premises that govern the RCTs logic and underpin their rigour. The lack of blinding, heterogeneity of participants, as well as poor treatment standardisation and difficulty of controlling for confounders, which characterise complex intervention RCTs, can potentially be profoundly detrimental to their integrity. Proponents of this approach argue that matching "real world" circumstances, while maintaining the randomised design, enhances external validity. We counter this argument by pointing out that an inverted U relation exists between internal and external validity, and thus relaxing the experimental conditions beyond a certain threshold can potentially paradoxically render the RCT externally invalid, i.e. its results cannot be used anywhere. We shall illustrate the inappropriate use of RCTs to evaluate community treatment orders and propose an alternative epistemic model that is based on mechanistic reasoning and Cartwright's capacity concept.

The Trouble with Evidence-Based Medicine

“…This problematic situation is partly maintained by the largely prevailing dogmatic approach within the EBM movement that prima facie prioritizes RCTs over credible, nonrandomized evidence even when the premises necessary to ensure the rigor of those RCTs cannot be reliably achieved, owing to factors such as complexity, severity or ethical considerations [6,7], and traditionally assigns undue primacy to heterogeneous meta-analyses that may have effectively ungeneralizable results [4]. There is no consensus as to the minimum level of homogeneity sufficient to render averaging data from different trials meaningful, and meta-analysts routinely proceed to pool data from individual trials even when detrimental levels of clinical and methodological heterogeneity exist, while misleadingly using random, rather than fixed, effect analysis model as a panacea [8].…”

“…What started a quarter of a century ago as a scientific and methodical approach to the practice and teaching of medicine risks being reduced into a form of post-expertise statistical medicine that has little relevance to real-life clinical practice. Psychiatry is particularly susceptible due to the frequent unrepresentativeness of patients recruited in RCTs [6] and remarkable heterogeneity and complexity of patient populations, interventions, and clinical settings, limiting the usefulness of meta-analyses [4,7].…”

“…Amid an opioid public health crisis, the World Health Organization has recently retracted its opioid prescribing guidelines due to concerns they were unduly influenced by financial interests [1], which demonstrates the troubled state of affairs of evidence-based medicine (EBM) [2] and challenges the untenable all-is-well stance of some of the main EBM centers [3]. A flood of randomized controlled trials (RCTs), systematic reviews, and meta-analyses (the workhorses of EBM) that are compromised by methodological flaws or financial or intellectual conflicts of interest [2,[4][5][6] may have inevitably influenced clinical practice, potentially exposing patients to iatrogenic morbidity and mortality, which paradoxically mirrors the concerns that originally drove the pioneers of EBM [3].…”

“…

would conventionally, alarmingly, occupy the top of the EBM pyramid [3].What started a quarter of a century ago as a scientific and methodical approach to the practice and teaching of medicine risks being reduced into a form of post-expertise statistical medicine that has little relevance to real-life clinical practice. Psychiatry is particularly susceptible due to the frequent unrepresentativeness of patients recruited in RCTs [6] and remarkable heterogeneity and complexity of patient populations, interventions, and clinical settings, limiting the usefulness of meta-analyses [4,7].I recommend that, in addition to guarding practice guidelines against financial conflicts of interest, EBM needs to broaden its notion of evidence beyond RCTs and their meta-analyses [6,7,10] and should revive the role of expertise and clinical experience in the design and interpretation of systematic reviews. Meta-analyses maybe useful when individual trials are underpowered but highly homogeneous; otherwise they are likely unnecessary, or uninterpretable and misleading to patients, clinicians, and policy makers.

Disclosure StatementThe author has no conflicts of interest to declare.Amid an opioid public health crisis, the World Health Organization has recently retracted its opioid prescribing guidelines due to concerns they were unduly influenced by financial interests [1], which demonstrates the troubled state of affairs of evidence-based medicine (EBM) [2] and challenges the untenable all-is-well stance of some of the main EBM centers [3].

…”

“…I recommend that, in addition to guarding practice guidelines against financial conflicts of interest, EBM needs to broaden its notion of evidence beyond RCTs and their meta-analyses [6,7,10] and should revive the role of expertise and clinical experience in the design and interpretation of systematic reviews. Meta-analyses maybe useful when individual trials are underpowered but highly homogeneous; otherwise they are likely unnecessary, or uninterpretable and misleading to patients, clinicians, and policy makers.…”