Quantifying representativeness in randomized clinical trials using machine learning fairness metrics

Abstract

Objective We help identify subpopulations underrepresented in randomized clinical trials (RCTs) cohorts with respect to national, community-based or health system target populations by formulating population representativeness of RCTs as a machine learning (ML) fairness problem, deriving new representation metrics, and deploying them in easy-to-understand interactive visualization tools. Materials and Methods We represent RCT cohort enrollment as random binary classification fairness problems, and then show how ML fairness metrics based on enrollment fraction can be efficiently calculated using easily computed rates of subpopulations in RCT cohorts and target populations. We propose standardized versions of these metrics and deploy them in an interactive tool to analyze 3 RCTs with respect to type 2 diabetes and hypertension target populations in the National Health and Nutrition Examination Survey. Results We demonstrate how the proposed metrics and associated statistics enable users to rapidly examine representativeness of all subpopulations in the RCT defined by a set of categorical traits (eg, gender, race, ethnicity, smoking status, and blood pressure) with respect to target populations. Discussion The normalized metrics provide an intuitive standardized scale for evaluating representation across subgroups, which may have vastly different enrollment fractions and rates in RCT study cohorts. The metrics are beneficial complements to other approaches (eg, enrollment fractions) used to identify generalizability and health equity of RCTs. Conclusion By quantifying the gaps between RCT and target populations, the proposed methods can support generalizability evaluation of existing RCT cohorts. The interactive visualization tool can be readily applied to identified underrepresented subgroups with respect to any desired source or target populations.

Planning and Monitoring Equitable Clinical Trial Enrollment Using Goal Programming

“…Our previously work on RCT representativeness metrics [14], derived from machine learning fairness metrics, is used to evaluate enrollment representativeness. These metrics have a lower threshold τ l and an upper threshold τ u .…”

“…To understand the proposed approach, we first introduce some terms from population representativeness. Representativeness metrics: Quantitative measures for disparities between a target population and an observed sample [14]. Ideally, subgroup sizes are equal to their proportions in target population.…”

“…All experiments use the two protected attributes race/ethnicity and gender to demonstrate the model's ability on multivariate cases. The representativeness measurement used is Log Disparity [14], which is introduced at the beginning of section II. For A2, additional information of enrolled SPRINT participants is needed since we want to improve the representativeness of protected subgroups through the rest of recruitment process based on current enrollees.…”

“…To fulfill the NIH Inclusion Policy and address the concerns of enrollment process mentioned before, we propose a multi-objective goal-programming (GP) model for equitable enrollment planning and monitoring based on our previous work about RCT representativeness metrics [14].…”

Recommendations on test datasets for evaluating AI solutions in pathology

“…For features represented as metadata (e.g., patient age, slide scanner, or diagnosis), bias can be detected by comparing the feature distributions in the test dataset and the target population using summary statistics (e.g., via mean and standard deviation) or dedicated fairness metrics [102,103]. Detection of bias in an entire test dataset requires a good estimate of the feature distribution of the target population of images.…”