A new improved graphical and quantitative method for detecting bias in meta-analysis

Abstract

Detection of publication and related biases remains suboptimal and threatens the validity and interpretation of meta-analytical findings. When bias is present, it usually differentially affects small and large studies manifesting as an association between precision and effect size and therefore visual asymmetry of conventional funnel plots. This asymmetry can be quantified and Egger's regression is, by far, the most widely used statistical measure for quantifying funnel plot asymmetry. However, concerns have been raised about both the visual appearance of funnel plots and the sensitivity of Egger's regression to detect such asymmetry, particularly when the number of studies is small. In this article, we propose a new graphical method, the Doi plot, to visualize asymmetry and also a new measure, the LFK index, to detect and quantify asymmetry of study effects in Doi plots. We demonstrate that the visual representation of asymmetry was better for the Doi plot when compared with the funnel plot. We also show that the diagnostic accuracy of the LFK index in discriminating between asymmetry due to simulated publication bias versus chance or no asymmetry was also better with the LFK index which had areas under the receiver operating characteristic curve of 0.74-0.88 with simulations of meta-analyses with five, 10, 15, and 20 studies. The Egger's regression result had lower areas under the receiver operating characteristic curve values of 0.58-0.75 across the same simulations. The LFK index also had a higher sensitivity (71.3-72.1%) than the Egger's regression result (18.5-43.0%). We conclude that the methods proposed in this article can markedly improve the ability of researchers to detect bias in meta-analysis.

Systematic review with meta‐analysis: anxiety and depression in children and adolescents with inflammatory bowel disease

“…Heterogeneity was assessed using the I 2 statistic, with values ≥75% indicating considerable heterogeneity . Reporting bias across studies (eg publication bias) was examined visually using “funnel plots” and the more sensitive “Doi plots” and formally using the Luis Furuya‐Kanamori (LFK) index, to see if the prevalence rates changed with increasing sample size. In the funnel plots and Doi plots, a higher prevalence is displayed by a higher “Double Arcsin Prevalence”, and a higher standard error indicates a lower sample size.…”

Review article: Emergency endotracheal intubation in non‐traumatic brain pathologies: A systematic review and meta‐analysis

“…We estimated the risk of publication bias by visually examining Doi plots and assessing the Luis Furuya‐Kanamori (LFK) index, rather than funnel plots . Funnel plots perform poorly when the effect size is a prevalence proportion .…”

“…Funnel plots perform poorly when the effect size is a prevalence proportion . The LFK index outperformed Egger's regression for detection of asymmetry because of biases caused by selective publication . The Doi plot uses a folded variant of the normal quantile‐ versus ‐effect plot.…”

“…Asymmetric Doi plots implies that small study effects (and related biases) may have affected the pooled effect estimate . The LFK index indicates no asymmetry if within ±1 unit, with values more extreme than ±1 unit implying asymmetry, but only if the asymmetry is in the same direction as the a priori judgement of the direction of suspected bias …”

Letter re: stratification by quality is not recommended in meta-analysis

“…We believe that bias occurs when studies are selected into strata such that systematic differences between larger and smaller studies emerge (where none initially existed), as shown in the funnel or Doi plot (Figures 2e4 in our published article [2]). The association develops solely because of selection of studies into a quality stratum and does not imply that all the results within strata defined by quality ranking are biased, and this is a situation akin to publication bias [3,4]. Page et al next state that within stratum bias would only be the case if results in smaller studies were biased and less precise.…”