False Dichotomies and Health Policy Research Designs: Randomized Trials Are Not Always the Answer

Abstract

Some medical scientists argue that only data from randomized controlled trials (RCTs) are trustworthy. They claim data from natural experiments and administrative data sets are always spurious and cannot be used to evaluate health policies and other population-wide phenomena in the real world. While many acknowledge biases caused by poor study designs, in this article we argue that several valid designs using administrative data can produce strong findings, particularly the interrupted time series (ITS) design. Many policy studies neither permit nor require an RCT for cause-and-effect inference. Framing our arguments using Campbell and Stanley’s classic research design monograph, we show that several “quasi-experimental” designs, especially interrupted time series (ITS), can estimate valid effects (or non-effects) of health interventions and policies as diverse as public insurance coverage, speed limits, hospital safety programs, drug abuse regulation and withdrawal of drugs from the market. We further note the recent rapid uptake of ITS and argue for expanded training in quasi-experimental designs in medical and graduate schools and in post-doctoral curricula.

Versorgungsforschung der Anti-VEGF-Therapie: Selektion und methodische Besonderheiten

“…Die Rahmenbedingungen können sich deutlich vom Alltag eines Gesundheitssystems unterscheiden [2]. Während die Zulassungsstudien des Depotsteroids Dexamethason eine Wiederbehandlung erst nach 6 Monaten vorsah [3], fand in einerebenfalls vom Hersteller initiiertenprospektiven Beobachtungsstudie die mittlere Wiederbehandlung bei Ödemen nach Astvenen-und Zentralvenenverschluss bereits nach 151 bzw.…”

What to do (or not to do) when randomization is not possible

“…Fortunately, not all nonrandomized studies are created equal, and strong quasiexperimental designs, when carefully used, can mimic randomization and produce valid findings. 4 Peng and colleagues use one such quasi-experimental design -regression discontinuity design -and find that patients assigned after their 18 th birthday to an adult allocation system had longer wait times and were less likely to receive a transplant than patients assigned before their 18 th birthday to a paediatric allocation system, despite no observed differences in waitlist-associated mortality between the groups.…”

Comment on ‘Measuring the impact of medicines regulatory interventions – systematic review and methodological considerations’ by Goedecke et al.

“…However, we were surprised that Goedecke et al included studies using weak research designs (for example, cross-sectional studies) that are typically excluded by international standards, i.e., Cochrane Systematic Reviews [3]. Secondly, their search and review omitted one of the most important studies in the field, our paper [4] published by BMJ in 2014 reporting an interrupted time series analysis of changes in antidepressant use and suicidal behaviour after safety FDA warnings from the US Food and Drug Administration (FDA) and media coverage.Weak research designs include single observations before and after (pre-post) an intervention without any controls and simple cross-sectional designs that correlate the exposure to an intervention with outcomes at a single point in time [5]. These designs are excluded by Cochrane systematic reviews of effects of health policies or programmes [6].…”

“…Weak research designs include single observations before and after (pre-post) an intervention without any controls and simple cross-sectional designs that correlate the exposure to an intervention with outcomes at a single point in time [5]. These designs are excluded by Cochrane systematic reviews of effects of health policies or programmes [6].…”

“…Given the complexity, use of rigorous study designs is crucial to establish cause and effect relationships. Reliable evidence based on well-designed studies can save lives [5]. Systematic reviews of effects of regulatory interventions should apply strict criteria to include only studies using rigorous designs that can produce unbiased results.…”