The Certainty Framework for Assessing Real‐World Data in Studies of Medical Product Safety and Effectiveness

Abstract

A fundamental question in using real‐world data for clinical and regulatory decision making is: How certain must we be that the algorithm used to capture an exposure, outcome, cohort‐defining characteristic, or confounder is what we intend it to be? We provide a practical framework to help researchers and regulators assess and classify the fit‐for‐purposefulness of real‐world data by study variable for a range of data sources. The three levels of certainty (optimal, sufficient, and probable) must be considered in the context of each study variable, the specific question being studied, the study design, and the decision at hand.

Development and Validation of Coding Algorithms to Identify Patients with Incident Non-Small Cell Lung Cancer in United States Healthcare Claims Data

“…Researchers should consider the algorithm’s performance in the context of their study question and data source, as described in the Certainty Framework for real-world data variables. 27 The implementation of a previously validated, broader lung cancer algorithm increased the performance of the point-based algorithm, which may be an important consideration for researchers building algorithms that comprise a subtype of the disease.…”

Quality assessment of real-world data

“…В то же время неоднородность выборки, отсутствие рандомизации и ослепления, характерные для исследований реальной клинической практики, создают сложности в статистической обработке полученных данных: какие бы сложные статистические методики ни использовались для исправления смещения выборки данных, полностью исключить влияние внешних факторов при их анализе не представляется возможным [4,5,6,14,15]. Имеющиеся сложности статистической обработки дополняются сложностями, связанными с интерпретацией исходных данных: наличие пропущенных данных, данных, представленных неструктурированно, без использования единых нормирующих шкал, препятствует отслеживанию динамики состояния пациента во времени [16].…”

HARmonized Protocol Template to Enhance Reproducibility of hypothesis evaluating real‐world evidence studies on treatment effects: A good practices report of a joint ISPE/ISPOR task force

“…ISPOR, [38][39][40][41][42] ISPE, 24,43,44 the FDA, 1,45,46 the EMA, 17,47 the European Network of Centres for Pharmacoepidemiology and Pharmacovigilance (ENCePP ® ), 5,[48][49][50] and the European Network for Health Technology Assessment (EUnetHTA) 51 and the Japanese Pharmaceuticals and Medical Device Agency (PMDA) 52 have all published guidance documents on good practice. Widely used checklists for the reporting of observational studies include RECORD-PE, 53 STROBE 54 and CHEERS.…”