Development of a multivariate prediction model to identify individual case safety reports which require clinical review

Abstract

Background The number of Individual Case Safety Reports (ICSRs) in pharmacovigilance databases are rapidly increasing world‐wide. The majority of ICSRs at the Netherlands Pharmacovigilance Centre Lareb is reviewed manually to identify potential signal triggering reports (PSTR) or ICSRs which need further clinical assessment for other reasons. Objectives To develop a prediction model to identify ICSRs that require clinical review, including PSTRs. Secondly, to identify the most important features of these reports. Methods All ICSRs (n = 30 424) received by Lareb between October 1, 2017 and February 26, 2021 were included. ICSRs originating from marketing authorisation holders and ICSRs reported on vaccines were excluded. The outcome was defined as PSTR (yes/no), where PSTR ‘yes’ was defined as an ICSR discussed at a signal detection meeting. Nineteen features were included, concerning structured information on: patients, adverse drug reactions (ADR) or drugs. Data were divided into a training (70%) and test set (30%) using a stratified split to maintain the PSTR/no PSTR ratio. Logistic regression, elastic net logistic regression and eXtreme Gradient Boosting models were trained and tuned on a training set. Random down‐sampling of negative controls was applied on the training set to adjust for the imbalanced dataset. Final models were evaluated on the test set. Model performances were assessed using the area under the curve (AUC) with 95% confidence interval of a receiver operating characteristic (ROC), and specificity and precision were assessed at a threshold for perfect sensitivity (100%, to not miss any PSTRs). Feature importance plots were inspected and a selection of features was used to re‐train and test model performances with fewer features. Results 1439 (4.7%) of reports were PSTR. All three models performed equally with a highest AUC of 0.75 (0.73–0.77). Despite moderate model performances, specificity (5%) and precision (5%) were low. Most important features were: ‘absence of ADR in the Summary of product characteristics’, ‘ADR reported as serious’, ‘ADR labelled as an important medical event’, ‘ADR reported by physician’ and ‘positive rechallenge’. Model performances were similar when using only nine of the most important features. Conclusions We developed a prediction model with moderate performances to identify PSTRs with nine commonly available features. Optimisation of the model using more ICSR information (e.g., free text fields) to increase model precision is required before implementation.

Natural language processing for automated triage and prioritization of individual case safety reports for case-by-case assessment

“…Current model with NLP performed better than our previously developed model that had an AUC of 0.75 (95% CI: 0.73-0.77). In addition, upon visual inspection of the precision and recall plot it was found that the current model also outperformed the previous model which was only based on structured fields (non-NLP features) (Gosselt et al, 2022). From the feature importance plot it can be seen that the feature 'info_length' seemed of high importance to distinguish case reports from non-case reports.…”

“…Most important features in this prediction model were: "absence of ADR in the Summary of product characteristics," "ADR reported as serious," "ADR labelled as an important medical event," "ADR reported by physician" and "positive rechallenge." An AUC of 0.75 (0.73-0.77) was obtained, which can be seen as moderate model performance (Gosselt et al, 2022).…”

Navigating duplication in pharmacovigilance databases: a scoping review

“…Moreover, systems ought to avoid over-reliance on automated standalone software programs as they could generate false signals. Instead, holistic methods which include manual review alongside automated processes should be preferred 33 34. The objective of this scoping review is to provide a comprehensive overview of duplication in pharmacovigilance databases on a global scale.…”

A Pilot, Predictive Surveillance Model in Pharmacovigilance Using Machine Learning Approaches

“…There are limited industry-wide standards that apply ML technologies in PV. Nonetheless, the number of ICSRs in PV databases are increasing, and given that individual case review is time-consuming, new approaches should be explored to help prioritize which reports require further evaluation by safety teams [ 4 ]. A previously published article revealed that ML algorithms have better accuracy in detecting new signals than traditional disproportionality analysis methods, which typically encounter some issues, such as background noise and the possibility of generating false-positive results [ 5 ].…”