Emulating Randomized Clinical Trials With Nonrandomized Real-World Evidence Studies

Abstract

Background: Regulators are evaluating the use of non-interventional real-world evidence (RWE) studies to assess the effectiveness of medical products. The RCT-DUPLICATE initiative uses a structured process to design RWE studies emulating randomized controlled trials (RCTs) and compare results. Here, we report findings of the first 10 trial emulations, evaluating cardiovascular outcomes of antidiabetic or antiplatelet medications. Methods: We selected 3 active-controlled and 7 placebo-controlled RCTs for replication. Using patient-level claims data from US commercial and Medicare payers, we implemented inclusion/exclusion criteria, selected primary endpoints, and comparator populations to emulate those of each corresponding RCT. Within the trial-mimicking populations, we conducted propensity score matching to control for >120 pre-exposure confounders. All study parameters were prospectively defined and protocols registered before hazard ratios (HRs) and 95% confidence intervals (CIs) were computed. Success criteria for the primary analysis were pre-specified for each replication. Results: Despite attempts to emulate RCT design as closely as possible, differences between the RCT and corresponding RWE study populations remained. The regulatory conclusions were equivalent in 6 of 10. The RWE emulations achieved a HR estimate that was within the 95% CI from the corresponding RCT in 8 of 10 studies. In 9 of 10, either the regulatory or estimate agreement success criteria were fulfilled. The largest differences in effect estimates were found for RCTs where second-generation sulfonylureas were used as a proxy for placebo regarding cardiovascular effects. Nine of 10 replications had a standardized difference between effect estimates of <2, which suggests differences within expected random variation. Conclusions: Agreement between RCT and RWE findings varies depending on which agreement metric is used. Interim findings indicate that selection of active comparator therapies with similar indications and use patterns enhances the validity of RWE. Even in the context of active comparators, concordance between RCT and RWE findings is not guaranteed, partially because trials are not emulated exactly. More trial emulations are needed to understand how often and in what contexts RWE findings match RCTs. Clinical Trial Registration: URL: https://clinicaltrials.gov Unique Identifiers: NCT03936049, NCT04215523, NCT04215536, NCT03936010, NCT03936036, NCT03936062, NCT03936023, NCT03648424, NCT04237935, NCT04237922

The use of nonrandomized evidence to estimate treatment effects in health technology assessment

“…A recent meta-epidemiological study conducted by IMPACT-HTA work package six consortium found no evidence of a systematic difference in treatment effects for pharmaceutical interventions but substantial variation in these differences for specific clinical questions. Similar results were observed among the first ten trial emulations from the RCT-DUPLICATE project in which RCTs were replicated in US claims databases [17]. There is stronger evidence that nonrandomized studies relying on external controls, such as single-arm trials, are associated with greater bias on average than other types of nonrandomized studies [16].…”

Association Between Glucagon-Like Peptide 1 Receptor Agonist and Sodium–Glucose Cotransporter 2 Inhibitor Use and COVID-19 Outcomes

“…Emerging evidence from the COVID-19 pandemic suggests that individuals with type 2 diabetes comprise a significant portion of the affected population and are at higher risk for severe outcomes including hospitalization and death (1,2). Due to the lack of a were selected as an active comparator because they are branded agents that have been well studied with minimal other clinical effects of concern, are among the five second-line therapies with prevalent use (17), and were recently suggested as an optimal comparator as a relatively newer agent among second-line therapies (18).…”

R WE ready for reimbursement? A round up of developments in real-world evidence relating to health technology assessment

“…A recent study from the randomized controlled trial (RCT)-DUPLICATE initiative reported the findings from the first ten trial emulations evaluating cardiovascular outcomes of antidiabetic or antiplatelet medications using RWD from three US healthcare claims data sources [3]. Patients were selected according to corresponding RCT inclusion/exclusion criteria, and end points similarly matched in claims data.…”

“…Given the body of evidence presented to regulators typically forms the core of that subsequently presented to HTA bodies and payers, it is vital that those involved in HTA/market access remain aware of these developments and proactively consider any potential impact for HTA. The outcomes from Franklin et al, however, may not translate directly into outcomes that would be important for HTA decisions [3]. For example, the measure of 'regulatory agreement' may not translate into a similar concept of 'HTA agreement', as HTA bodies are more concerned with the magnitude of effect and implications for cost-effectiveness estimates rather than simply the existence and direction of an effect.…”