Inferential Challenges for Real-world Evidence in the Era of Routinely Collected Health Data

Abstract

Routinely collected health data (RCHD) from administrative claims, electronic medical records, disease registries, and other sources transform the way in which knowledge about cancer is generated. Created and collected as part of routine oncological practice, these data capture cancer-related care provided to real-world patients in real-world settings. As with other types of realworld data, including data generated by patients through websites and wearable devices, measures of social determinants of health, and environmental exposures, RCHD are increasingly popular among researchers, funders, and policy makers because of the potential to generate actionable evidence. The availability of RCHD at low or no cost for large patient populations with long followed-up periods and almost real-time updates has attracted numerous researchers pursuing a nearly infinite number of research questions using any particular set of RCHD.The overarching goal of RCHD is to improve care and outcomes for patients with cancer, a goal that can only be achieved when analyses provide valid inferences. For years, central to the validity of RCHD studies were issues related to data quality (eg, the ability of RCHD to accurately capture clinical information) and biases originating from the observational nonrandomized nature of these studies (eg, confounding). Thankfully, with advances in informatics and causal inference, there is now an array of approaches to overcome the caveats. However, 2 other distinct properties of RCHD have received little attention despite their potential to undermine the validity of inferences: (1) the use of RCHD for formulating research hypotheses that are then examined with the same data (selective inference), and (2) the pursuing of numerous research questions within the same database (multiple testing).In this Viewpoint, we discuss how biases originating from selective inference and multiple testing can potentially invalidate conclusions from RCHD studies, even when high-quality data are analyzed with state-ofthe-art causal inference methodologies.

Impact of two waves of Sars‐Cov2 outbreak on the number, clinical presentation, care trajectories and survival of patients newly referred for a colorectal cancer: A French multicentric cohort study from a large group of university hospitals

“…Nonetheless, our results on new cancer cases in Paris region mirror a national independent analysis of CRC care during the pandemic 4 . Those results should be interpreted in the context of routinely collected medical data with relevant information being missing such as the surgery quality assessment 46 . We included tertiary care centers with high‐volume cancer surgeries which is associated with better patients' outcomes 47 .…”

“… 4 Those results should be interpreted in the context of routinely collected medical data with relevant information being missing such as the surgery quality assessment. 46 We included tertiary care centers with high‐volume cancer surgeries which is associated with better patients' outcomes. 47 No particular method was implemented to account for missing textual data.…”

Harnessing Real-World Evidence to Advance Cancer Research

“…Pre-registration of protocols is a standard requirement for conducting RCTs, and this approach is increasingly being adopted for real-world data research to reduce the risk of multiple hypotheses testing within studies, improve methodological transparency and support replication efforts [69,79]. Pre-registration can also help to address the communitylevel multiplicity issues that arise when multiple hypotheses are cumulatively tested by different researchers within a real-world database, by minimizing the effects of selective reporting and allowing researchers to identify all previously tested hypotheses [86]. ISPOR-ISPE have recently launched a Real-World Evidence Registry to facilitate pre-registration of real-world research protocols and promote trust in study results [87].…”

Reproducibility of prediction models in health services research

“…Data sources in health services research often differ from the traditional epidemiological investigations that prospectively or retrospectively collect data through active recruitment of participants based on a priori specified research questions [ 2 ]. Indeed, health services studies heavily rely on data that are routinely collected for purposes other than research, including health care billing claims, registry data, or electronic health records [ 1 , 3 ]. Outcomes frequently examined by health services researchers include health care spending, and utilization of care services (e.g., hospital admission or readmission, admission to intensive care unit, length of hospitalization, or emergency department visit).…”