Raising the Bar for Real-World Data in Oncology: Approaches to Quality Across Multiple Dimensions

Abstract

PURPOSE Electronic health record (EHR)–based real-world data (RWD) are integral to oncology research, and understanding fitness for use is critical for data users. Complexity of data sources and curation methods necessitate transparency into how quality is approached. We describe the application of data quality dimensions in curating EHR-derived oncology RWD. METHODS A targeted review was conducted to summarize data quality dimensions in frameworks published by the European Medicines Agency, The National Institute for Healthcare and Excellence, US Food and Drug Administration, Duke-Margolis Center for Health Policy, and Patient-Centered Outcomes Research Institute. We then characterized quality processes applied to curation of Flatiron Health RWD, which originate from EHRs of a nationwide network of academic and community cancer clinics, across the summarized quality dimensions. RESULTS The primary quality dimensions across frameworks were relevance (including subdimensions of availability, sufficiency, and representativeness) and reliability (including subdimensions of accuracy, completeness, provenance, and timeliness). Flatiron Health RWD quality processes were aligned to each dimension. Relevancy to broad or specific use cases is optimized through data set size and variable breadth and depth. Accuracy is addressed using validation approaches, such as comparison with external or internal reference standards or indirect benchmarking, and verification checks for conformance, consistency, and plausibility, selected on the basis of feasibility and criticality of the variable to the intended use case. Completeness is assessed against expected source documentation; provenance by recording data transformation, management procedures, and auditable metadata; and timeliness by setting refresh frequency to minimize data lags. CONCLUSION Development of high-quality, scaled, EHR-based RWD requires integration of systematic processes across the data lifecycle. Approaches to quality are optimized through knowledge of data sources, curation processes, and use case needs. By addressing quality dimensions from published frameworks, Flatiron Health RWD enable transparency in determining fitness for real-world evidence generation.

Data Quality–Driven Improvement in Health Care: Systematic Literature Review

“…In an exploration of the DQ literature, we found 10 other reviews that summarise the most frequently represented DQ dimensions across DQ studies, software, and theoretical frameworks [19,21,[23][24][25][26][27][28][29][30]. All 10 reviews demonstrated the concept of completeness to be well represented.…”

“…The need for high-quality data was exemplified during the COVID-19 pandemic, when EHR-RWD was critical for research and planning [15]. Regulatory authorities, such as the FDA (Food and Drug Administration), EMA (European Medicines Agency), NICE (National Institute for Health and Care Excellence), and MHRA (Medicines and Healthcare products Regulatory Agency), recommend the reporting of DQ metrics and dimensions to provide additional context to real-world study outcomes, therefore serving as the foundation for trustworthy RWE [3,14,[16][17][18][19]. These guidelines mostly promote ad-hoc DQ assessment and reporting, with the exception that EMA briefly notes the importance of assessing DQ as close as possible to the moment of data capture to help with collection errors [3,[16][17][18].…”

When Will Real-World Data Fulfill Its Promise to Provide Timely Insights in Oncology?

“…25 The need for transparency was emphasized in an editorial written by the FDA and published in this journal. 26 In the same issue, an assessment of DQ in one oncology RWD source provides insight into the challenges in creating a fitfor-purpose oncology RWD source. 27 The evolving consensus among stakeholders appears to agree on the following elements required to produce credible RWE from nonrandomized noninterventional studies.…”

The Bar Is High: Evaluating Fit-for-Use Oncology Real-World Data for Regulatory Decision Making

“…Audit trails starting at data extraction and extending through maintenance and retention that include access to source records or certified copies can help promote transparency and reproducibility. The analysis by Castellanos et al 4 highlights the importance of transparency as a means to characterize fitness-for-use of a singular specific RWD source. However, this high-level assessment of fitness-for-use is limited absent a specific research question.…”

“…In addition, pragmatic trials may incorporate RWD to minimize clinician and patient burden and streamline design while maintaining randomization, with the potential to generate evidence for new indications and evaluate safety and effectiveness in the postmarketing setting. 16 Finally, as alluded to by Castellanos et al, 4 a vast amount of unstructured RWD for oncology are becoming available including radiology reports, laboratory reports, physician notes, and genomic sequencing data. Whereas manual abstraction of such data is labor intensive, the application of artificial intelligence (AI), including techniques such as natural language processing, large language models, and machine learning, holds promise as a means to facilitate data abstraction.…”