Diagnostics for generalized linear hierarchical models in network meta‐analysis

Abstract

Network meta-analysis (NMA) combines direct and indirect evidence comparing more than 2 treatments. Inconsistency arises when these 2 information sources differ. Previous work focuses on inconsistency detection, but little has been done on how to proceed after identifying inconsistency. The key issue is whether inconsistency changes an NMA's substantive conclusions. In this paper, we examine such discrepancies from a diagnostic point of view. Our methods seek to detect influential and outlying observations in NMA at a trial-by-arm level. These observations may have a large effect on the parameter estimates in NMA, or they may deviate markedly from other observations. We develop formal diagnostics for a Bayesian hierarchical model to check the effect of deleting any observation. Diagnostics are specified for generalized linear hierarchical NMA models and investigated for both published and simulated datasets. Results from our example dataset using either contrast- or arm-based models and from the simulated datasets indicate that the sources of inconsistency in NMA tend not to be influential, though results from the example dataset suggest that they are likely to be outliers. This mimics a familiar result from linear model theory, in which outliers with low leverage are not influential. Future extensions include incorporating baseline covariates and individual-level patient data.

Power and Commensurate Priors for Synthesizing Aggregate and Individual Patient Level Data in Network Meta-Analysis

“…Later, Zhao et al . () introduced diagnostic methods to detect influential observations that might cause inconsistency and studied the effect of these influential observations on conclusions drawn from an NMA. However, further methodological development is required to apply these approaches to our analysis because it combines IPD and AD by assigning different prior weights to different sources of evidence.…”

“…Recently, Zhao et al (2016) have proposed methods to detect inconsistency in arm-based NMA models with only AD studies by computing discrepancy factors and then identifying trials that are the sources of inconsistency by investigating extreme study-specific random-effect estimates. Later, Zhao et al (2017) introduced diagnostic methods to detect influential observations that might cause inconsistency and studied the effect of these influential observations on conclusions drawn from an NMA. However, further methodological development is required to apply these approaches to our analysis because it combines IPD and AD by assigning different prior weights to different sources of evidence.…”

The statistical importance of a study for a network meta-analysis estimate

“…We emphasize that the importance of a study is always conditional on the other studies being included in the network. There is an analogy to a multivariable regression model: If the association of each regressor (covariate) x i with the dependent variable is considered in isolation, the proportion of explained variance of the dependent variable is given by its coefficient of determination, r 2 i , which is bounded between 0 and 1. When considering more than one covariate, it does not make sense to consider their r 2 i values separately (or even to add them).…”

“…For those mainly interested in treatment effects, we point to approaches to identify influential studies (or 'outliers') which impact the effect estimates [1,2,3]. These concepts do not rely exclusively on the structure of the network and the variances of the studies, but also account for their effect estimates and the extent to which they are consistent with estimates from other studies.…”

“…To describe the influence of a study on a MA estimate, we distinguish between approaches that focus on the study's contribution in the sense of attributing a 'weight' to each study, mainly depending on the variances, and approaches that also take into account the direction and size of each treatment effect, and which are seeking for 'influential' or explicitly 'outlying' studies. For the latter, see [1,2,3]. In this paper, we concentrate on the first approach; that is, we are mainly interested in measuring study contributions without looking at their effect estimates.…”