Estimating the effect of adjuvant chemo-therapy for colon-cancer using registry data: a method comparison and validation

Abstract

Background: Although randomized controlled trials (RCT) are the gold standard to estimate treatment effects, they are often criticized in terms of generalizability. Observational data might alleviate this problem by being readily available in large quantities. However, observational data are potentially confounded. In this methodological study we use a large-scale RCT as the gold standard to examine the performance of various statistical methods to control for potential confounding in observational data. Methods: In this paper we compare three types of methods that allow researchers to correct for such potential confounding: direct methods, inverse probability weighting (IPW) methods and doubly robust (DR) methods. We uniquely compare estimates obtained from the population-wide Netherlands Cancer Registry (NCR) on colon cancer (n=52621) with estimates obtained from a large-scale RCT. As the RCT differs from the observational data both in its sampling mechanism and in its treatment assignment mechanism, we first resample the NCR data to reflect the distribution in RCT data. Next, we correct for potential confounding using three alternative types of methods and consequentially evaluate their estimates to those obtained in the RCT. Results: We find that while all estimators qualitatively approximate to findings in the RCT, methods that can flexibly model the response (i.e., direct estimation and DR estimation) performed consistently superior to the inverse propensity score method. Subgroup analysis indicates that relatively simple models allow us to properly estimate the treatment effect. However, these simple models do not properly identify heterogeneous treatment effects in stage2 colon cancer. Careful sensitivity analysis using more flexible models demonstrates both the uncertainty and the potential heterogeneous treatment effect in stage2 cancer and provides robust estimation of treatment effect in stage3 cancer. Conclusions: Our results suggest that both the direct method and the DR method, when executed with care, can be used to reliably estimate treatment effects based on registry data. This methodological validation opens the door to more extensive use of registry data for the estimation of (individual) treatment effects. Additionally, our identification of potentially meaningful subgroups of stage2 colon cancer patients who, based on our analysis seem to benefit from chemotherapy, should be further explored.

Value of Real-World Evidence for Treatment Selection: A Case Study in Colon Cancer

“…The method regresses the conditional probability of 5-year OS, given patient and tumor characteristics, treatments, and estimates of the propensity score. 24 As is shown in a comparative study based on the same population, 25 the best-performing method for this use case—where best is defined in terms of correspondence to existing RCT data 26 —uses a Bayesian additive regression tree to model both the probability of the outcome and the propensity scores. The following variables were adjusted in both models: pN, pT, grade, EMVI, IMVI, lymphatic invasion, the number of lymph nodes examined, age at diagnosis, sex, ASA physical status, year of diagnosis, tumor location, colon perforation, and MSI status.…”

RCTrep: An R Package for the Validation of Estimates of Average Treatment Effects

“…Algorithm 1 Overview of the approach to derive the interpretable treatment rules [16,17], where w i is an estimate of weight for individual i to adjust for the covariate mismatch between two datasets, L is a distance measurement. If data from a RCT is not available, we can select f using validation methods in [26,27,28,2].…”