Clinical Trial Generalizability Assessment in the Big Data Era: A Review

Abstract

Clinical studies, especially randomized, controlled trials, are essential for generating evidence for clinical practice. However, generalizability is a long‐standing concern when applying trial results to real‐world patients. Generalizability assessment is thus important, nevertheless, not consistently practiced. We performed a systematic review to understand the practice of generalizability assessment. We identified 187 relevant articles and systematically organized these studies in a taxonomy with three dimensions: (i) data availability (i.e., before or after trial ( a priori vs. a posteriori generalizability)); (ii) result outputs (i.e., score vs. nonscore); and (iii) populations of interest. We further reported disease areas, underrepresented subgroups, and types of data used to profile target populations. We observed an increasing trend of generalizability assessments, but < 30% of studies reported positive generalizability results. As a priori generalizability can be assessed using only study design information (primarily eligibility criteria), it gives investigators a golden opportunity to adjust the study design before the trial starts. Nevertheless, < 40% of the studies in our review assessed a priori generalizability. With the wide adoption of electronic health records systems, rich real‐world patient databases are increasingly available for generalizability assessment; however, informatics tools are lacking to support the adoption of generalizability assessment practice.

How the clinical research community responded to the COVID-19 pandemic: An analysis of the COVID-19 clinical studies in ClinicalTrials.gov

“…patients. As results of COVID-19 studies become available, we will be able to assess the extent to which the trial design and eligibility criteria in particular would impact the findings as well as the real-world population representativeness of these studies using generalizability assessment methods [6].…”

“…In future work, we will perform a longitudinal analysis of COVID-19 studies to assess the changes in the use of eligibility criteria and consideration of risk factors for severe illness in COVID-19 patients. As results of COVID-19 studies become available, we will be able to assess the extent to which the trial design and eligibility criteria in particular would impact the findings as well as the real-world population representativeness of these studies using generalizability assessment methods [ 6 ].…”

“…Nonetheless, conducting COVID-19 clinical studies could still be challenging in the traditional clinical trial eco-system, where patient accrual is often delayed due to logistical constraints [21]. The generalizability of the study results to the real-world population should be evaluated with state-of-the-art techniques [6]. Older adults could have still been underrepresented in COVID-19 clinical studies due to logistical reasons, which can only be assessed with the published results after the completion of the studies [22].…”

Using Real-World Data to Rationalize Clinical Trials Eligibility Criteria Design: A Case Study of Alzheimer’s Disease Trials

“…To date, a number of methods and tools have been developed to quantify clinical trials' population representativeness (or generalizability). 9 These methods can be categorized into two major approaches: (1) sample-driven and often called a posterior generalizability, where these methods measure the representativeness the study samples (i.e., participants enrolled in clinical trials) over the target population, and (2) eligibility-driven and called a priori a Qian Li, MS and Yi Guo, PhD contributed equally, co-first authors b Corresponding: Jiang Bian, PhD; bianjiang@ufl.edu generalizability, where these methods measure the representativeness of the study population (i.e., patients who met the eligibility criteria) over the target population. 10 Although the a posterior generalizability is important, it cannot be changed after the fact as the trial has already been concluded.…”

Deep Learning Approach to Parse Eligibility Criteria in Dietary Supplements Clinical Trials Following OMOP Common Data Model

“…16 Moreover, more than 70% of clinical trial generalizability assessment studies reported low generalizability of completed trials, partly due to low enrollment. 17 The rapid growth of the electronic health records (EHR) provides an unprecedented opportunity to harness its data to full potential for secondary use. 18 Moreover, the last few years have also witnessed an increasing number of clinical research networks focused on building large collections of data from EHRs and claims to provide cohort discovery services.…”