Improved Detection Criteria for Detecting Drug-Drug Interaction Signals Using the Proportional Reporting Ratio

Abstract

There is a current demand for “safety signal” screening, not only for single drugs but also for drug-drug interactions. The detection of drug-drug interaction signals using the proportional reporting ratio (PRR) has been reported, such as through using the combination risk ratio (CRR). However, the CRR does not consider the overlap between the lower limit of the 95% confidence interval of the PRR of concomitant-use drugs and the upper limit of the 95% confidence interval of the PRR of single drugs. In this study, we proposed the concomitant signal score (CSS), with the improved detection criteria, to overcome the issues associated with the CRR. “Hypothetical” true data were generated through a combination of signals detected using three detection algorithms. The signal detection accuracy of the analytical model under investigation was verified using machine learning indicators. The CSS presented improved signal detection when the number of reports was ≥3, with respect to the following metrics: accuracy (CRR: 0.752 → CSS: 0.817), Youden’s index (CRR: 0.555 → CSS: 0.661), and F-measure (CRR: 0.780 → CSS: 0.820). The proposed model significantly improved the accuracy of signal detection for drug-drug interactions using the PRR.

Verification of the “Upward Variation in the Reporting Odds Ratio Scores” to Detect the Signals of Drug–Drug Interactions

“…The drugs registered as "suspected drugs" in the verification dataset were considered as the drugs to be investigated. As in previous studies [16,18,[24][25][26], the only adverse event targeted for this signal search was set to be the preferred term; that is, "Stevens-Johnson syndrome" (SJS) in the Medical Dictionary for Regulatory Activities Japanese version (MedDRA/J), which is registered in the verification dataset.…”

“…If such an overlap occurs, a risk signal may not be detected for concomitant use. Therefore, with reference to the interaction signal score (INTSS) [27] and concomitant signal score (CSS) [26], we used the criteria to detect if "(ROR025 drug D1 ∩ drug D2) > 1" and "the lower limit of the 95%CI of the ROR of the combination of drug D1 and drug D2 (ROR025 drug D1 ∩ drug D2) is greater than either the upper limit of the 95% CI of drug D1 (ROR975 drug D1) or the upper limit of the 95% CI of the ROR of drug D2 (ROR975 drug D2), whichever is greater (Model 2)" (Figure 2, Equation ( 10)).…”

“…If such an overlap occurs, a risk signal may not be detected for concomitant use. Therefore, with reference to the interaction signal score (INTSS) [27] and concomitant signal score (CSS) [26], we used the criteria to detect if "(ROR 025 drug D1 ∩ drug D2 ) > 1" and "the lower limit of the 95% CI of the ROR of the combination of drug D 1 and drug D 2 (ROR 025 drug D1 ∩ drug D2 ) is greater than either the upper limit of the 95% CI of drug D 1 (ROR 975 drug D1 ) or the upper limit of the 95% CI of the ROR of drug D 2 (ROR 975 drug D2 ), whichever is greater (Model 2)" (Figure 2, Equation ( 10 If such an overlap occurs, a risk signal may not be detected for concomitant use. Therefore, with reference to the interaction signal score (INTSS) [27] and concomitant signal score (CSS) [26], we used the criteria to detect if "(ROR025 drug D1 ∩ drug D2) > 1" and "the lower limit of the 95%CI of the ROR of the combination of drug D1 and drug D2 (ROR025 drug D1 ∩ drug D2) is greater than either the upper limit of the 95% CI of drug D1 (ROR975 drug D1) or the upper limit of the 95% CI of the ROR of drug D2 (ROR975 drug D2), whichever is greater (Model 2)" (Figure 2, Equation ( 10)).…”

“…Although Susuta et al were the first to propose a detection method that uses the "upward variation in the ROR score" as the signals for drug-drug interactions in Japan [18], the proposed cutoff values of the scores and the comparison with other methods for detecting signals of drug-drug interactions have not been sufficiently verified, and the validity of the signals detected by this analysis method is unknown. Therefore, in this study, we aimed to verify an analysis method that uses the "upward variation in the ROR scores" as the signals for drug-drug interactions, referring to similar previous studies that evaluated the detection tendency of search algorithms [16,[24][25][26].…”

Spotlight commentary: The value of spontaneous reporting systems to detect (the lack of) clinically relevant drug–drug interactions in clinical practice

“…2,6 Different approaches have been explored and validated in the recent past, including the Ω shrinkage measure, the upward variation of disproportionality measures (e.g., the reporting odds ratio), and additive/multiplicative methods. 5,[7][8][9][10][11][12] Remarkably, no matter the complexity of the implemented analysis/method, the "signal" will never be more than a hypothesis and remains an alert rather than an alarm. Moreover, there is no consensus on the best disproportionality approach, thus leaving researchers with the challenge of choosing the algorithm likely to achieve the best performance in terms of accuracy (false positives vs false negatives), while taking into account expected reporting biases.…”