Real world evidence in cardiovascular medicine: ensuring data validity in electronic health record-based studies

Abstract

Objective With growing availability of digital health data and technology, health-related studies are increasingly augmented or implemented using real world data (RWD). Recent federal initiatives promote the use of RWD to make clinical assertions that influence regulatory decision-making. Our objective was to determine whether traditional real world evidence (RWE) techniques in cardiovascular medicine achieve accuracy sufficient for credible clinical assertions, also known as “regulatory-grade” RWE. Design Retrospective observational study using electronic health records (EHR), 2010–2016. Methods A predefined set of clinical concepts was extracted from EHR structured (EHR-S) and unstructured (EHR-U) data using traditional query techniques and artificial intelligence (AI) technologies, respectively. Performance was evaluated against manually annotated cohorts using standard metrics. Accuracy was compared to pre-defined criteria for regulatory-grade. Differences in accuracy were compared using Chi-square test. Results The dataset included 10 840 clinical notes. Individual concept occurrence ranged from 194 for coronary artery bypass graft to 4502 for diabetes mellitus. In EHR-S, average recall and precision were 51.7% and 98.3%, respectively and 95.5% and 95.3% in EHR-U, respectively. For each clinical concept, EHR-S accuracy was below regulatory-grade, while EHR-U met or exceeded criteria, with the exception of medications. Conclusions Identifying an appropriate RWE approach is dependent on cohorts studied and accuracy required. In this study, recall varied greatly between EHR-S and EHR-U. Overall, EHR-S did not meet regulatory grade criteria, while EHR-U did. These results suggest that recall should be routinely measured in EHR-based studes intended for regulatory use. Furthermore, advanced data and technologies may be required to achieve regulatory grade results.

An Electronic Health Record Text Mining Tool to Collect Real‐World Drug Treatment Outcomes: A Validation Study in Patients With Metastatic Renal Cell Carcinoma

“…However, we set thresholds for both precision and recall at 90%, to limit the chance on incorrect conclusions when data are used for treatment evaluation. This is in line with thresholds set by Hernandez‐Boussard et al 24 . Because a part of the IMDC criteria are measurement values, with the answer being a binary question, these will also be analyzed by calculating precision, recall, and F1‐score.…”

Glycemic Control Following GLP-1 RA or Basal Insulin Initiation in Real-World Practice: A Retrospective, Observational, Longitudinal Cohort Study

“…In both datasets, data were collected from clinical practice and were not specifically designed for research purposes. As a result, often data that would be useful for analysis may be missing, erroneous, or misclassified [22]. Moreover, dosing information of BI was not available, and individual glycemic control goals were not available.…”

<p>Harnessing Real-World Data for Regulatory Use and Applying Innovative Applications</p>

“…To support these efforts, AI capabilities are increasingly being applied to the analysis of RWD. 24,28 Furthermore, in a 2018 survey, 60% of pharma industry respondents were using AI in their RWE programs and 95% anticipated utilizing AI for this purpose in the coming years. 24 AI constitutes a combination of self-learning capabilities that mimic the way the human brain works, with the intent of replicating human decision-making and interactions.…”

“…However, researchers must first ensure that the data obtained are complete and relevant to the condition, patient population, and treatment analyzed. 28 For example, unstructured data may contain relevant information for only certain sub-populations or information may be entered for some patients but not others. Even structured data pose challenges in the application of AI to RWD, as field-entry data may be entered using inconsistent terms, may be formatted differently between sources, or may be incomplete or contain errors.…”

“…Algorithms that are not optimized might identify "false-positive" and "falsenegative" data, for example by including or excluding patients in analysis who do not actually meet the intended criteria. 28 A recent retrospective study of EHR data for patients with cardiovascular disease illustrates the difficulty that analysts encounter in algorithm development. The study results suggested that, despite the researchers taking various approaches toward generating the algorithms, the final model did not accurately analyze structured data and medication information was not sufficiently captured from unstructured data.…”