Treatment effects in randomised trials using routinely collected data for outcome assessment versus traditional trials: meta-research study

Abstract

Objective To compare effect estimates of randomised clinical trials that use routinely collected data (RCD-RCT) for outcome ascertainment with traditional trials not using routinely collected data. Design Meta-research study. Data source Studies included in the same meta-analysis in a Cochrane review. Eligibility criteria for study selection Randomised clinical trials using any type of routinely collected data for outcome ascertainment, including from registries, electronic health records, and administrative databases, that were included in a meta-analysis of a Cochrane review on any clinical question and any health outcome together with traditional trials not using routinely collected data for outcome measurement. Review methods Effect estimates from trials using or not using routinely collected data were summarised in random effects meta-analyses. Agreement of (summary) treatment effect estimates from trials using routinely collected data and those not using such data was expressed as the ratio of odds ratios. Subgroup analyses explored effects in trials based on different types of routinely collected data. Two investigators independently assessed the quality of each data source. Results 84 RCD-RCTs and 463 traditional trials on 22 clinical questions were included. Trials using routinely collected data for outcome ascertainment showed 20% less favourable treatment effect estimates than traditional trials (ratio of odds ratios 0.80, 95% confidence interval 0.70 to 0.91, I 2 =14%). Results were similar across various types of outcomes (mortality outcomes: 0.92, 0.74 to 1.15, I 2 =12%; non-mortality outcomes: 0.71, 0.60 to 0.84, I 2 =8%), data sources (electronic health records: 0.81, 0.59 to 1.11, I 2 =28%; registries: 0.86, 0.75 to 0.99, I 2 =20%; administrative data: 0.84, 0.72 to 0.99, I 2 =0%), and data quality (high data quality: 0.82, 0.72 to 0.93, I 2 =0%). Conclusions Randomised clinical trials using routinely collected data for outcome ascertainment show smaller treatment benefits than traditional trials not using routinely collected data. These differences could have implications for healthcare decision making and the application of real world evidence.

High Variability of COVID-19 Case Fatality Rate in Germany

“…Taken together, the results show a good agreement between the clinical and epidemiological approach (37) and provide further evidence that the high CFR in some German hospitals is caused by overtreatment (2), (38). As randomized clinical trials are largely impossible, also routinely collected data (39) are sufficient proof of the clinical observation that less invasive measures like HFNC could reduce the undoubtedly high CFR in ventilated COVID-19 patients. F. Distribution of COVID-19 CFR and sex by German states.…”

Reporting transparency and completeness in trials: Paper 1: Introduction - Better reporting for disruptive clinical trials using routinely collected data

“…Finally, trials conducted using existing cohorts or RCD may be able to respond more rapidly and nimbly to critical research needs, evidenced during the COVID-19 pandemic, though they also introduce complexity. Some studies use hybrid approaches that mix traditional tailored data collection with the use of RCD 7 . A recent example is the RECOVERY Trial, which utilizes routinely collected administrative and healthcare data from the UK NHS system for short-to medium-term outcome assessment, combined with short-term custom-built data collection to evaluate treatments that may be beneficial for people hospitalized with suspected or confirmed COVID-19 8 .…”

Comparative effectiveness and safety of pharmaceuticals assessed in observational studies compared with randomized controlled trials

“…The median ratio of the relative effect measure from observational studies to RCTs was 0.92, indicating just slightly lower effectiveness/safety estimates in observational studies than corresponding RCTs. This is in fact somewhat higher than the 0.80 ratio recently found in meta-research comparing effect estimates of randomized clinical trials that use routinely collected data (i.e., from traditional observational study sources such as registries, electronic health records, or administrative claims) for outcome ascertainment with traditional trials not using routinely collected data [ 47 ]. However, whether judging by the frequency of “extreme” differences (43.2%) or statistically significant differences in opposite directions (17.6%), one could not claim that observational study results consistently replicated RCT results on a study-by-study basis in our sample.…”