Outlier detection and influence diagnostics in network meta‐analysis

Abstract

Network meta-analysis has been gaining prominence as an evidence synthesis method that enables the comprehensive synthesis and simultaneous comparison of multiple treatments. In many network meta-analyses, some of the constituent studies may have markedly different characteristics from the others, and may be influential enough to change the overall results. The inclusion of these "outlying" studies might lead to biases, yielding misleading results. In this article, we propose effective methods for detecting outlying and influential studies in a frequentist framework. In particular, we propose suitable influence measures for network meta-analysis models that involve missing outcomes and adjust the degree of freedoms appropriately. We propose three influential measures by a leave-one-trial-out cross-validation scheme: (1) comparison-specific studentized residual, (2) relative change measure for covariance matrix of the comparative effectiveness parameters, (3) relative change measure for heterogeneity covariance matrix. We also propose (4) a model-based approach using a likelihood ratio statistic by a mean-shifted outlier detection model. We illustrate the effectiveness of the proposed methods via applications to a network meta-analysis of antihypertensive drugs. Using the four proposed methods, we could detect three potential influential trials involving an obvious outlier that was retracted because of data falsifications. We also demonstrate that the overall results of comparative efficacy estimates and the ranking of drugs were altered by omitting these three influential studies. K E Y W O R D S contrast-based model, influence diagnostics, multivariate meta-analysis, network meta-analysis, outlier detection 1 | INTRODUCTION Network meta-analysis has been gaining prominence as an evidence synthesis method that enables the comprehensive synthesis and simultaneous comparison of multiple treatments. The methodology can synthesize both direct and indirect evidence for all treatment comparisons of interest, and provides estimates of comparative efficacies among treatments, even when no direct comparison evidence exists for some of the included treatments. 1,2 In network meta-analysis, there are often systematic heterogeneities among the synthesized studies, for example, study designs, participant characteristics, regions, sites, treatment administration, interventions, outcome definitions, and so on. Therefore, heterogeneity of the

Psychotherapies for depression: a network meta‐analysis covering efficacy, acceptability and long‐term outcomes of all main treatment types

“…Because of the global inconsistency in the network, we searched for the sources of trial‐level influential factors by a bootstrapping method 25 . Through the bootstrap‐based evaluation, 37 trials were detected as influential outliers.…”

Exploring methodological challenges in network meta-analysis models and developing methodology for outlier detection

“…Following bibliographic recommendations, several cut-offs for the detection measures are provided but this should not strictly be used as this offered as empirical rules to make conclusions for outlyingness and influential cases. For example, Viechtbauer and Cheung [85] provided values 1.96 and 2 for the absolute studentized residuals while Noma et al [88] following a parametric bootstrap method to obtain the sampling distribution for studentized residual. Τhere is no subjective rule in the diagnosis of outlyingness as conclusions made due to sharp changes or empirical cut-offs in proposed measures.…”

“…Viechtbauer and Cheung propose the ratio of the determinants of the variance-covariance matrix of treatment estimates (COVRATIO) when excluding the ℎ study from model fitting [85]. Noma et al [88] extended the COVRATIO measure in a multivariate meta-regression model. Hedges and Olkin suggested also to examine changes in Cochran's statistic [84].…”

“…Based on the formula that Viechtbauer and Cheung provided for Cook's statistic in meta-analytical context. Noma et al[88] extended the Cook's distance measure in a multivariate meta-regression model. The analogy for the Cook's distance or Cook's statistic…”

“…Zhang et al[86] provided four measures for the detection of outliers in the Bayesian hierarchical model while Zhao et al[87] offered outlier detection measures for generalized Bayesian hierarchical models to detect outliers at observation data and not at aggregated data. Noma et al[88] have recently provided four measures to detect outliers in frequentist NMA model based on multivariate random-effects meta-regression method.In this Chapter, we provide a brief synopsis of the several outlier detection strategies that have been proposed in regression and meta-analytical context. We propose and evaluate several measures and graphical tools that seek to accomodate influential studies and outliers in network meta-analysis.…”