Robust variance estimation in meta‐regression with binary dependent effects

Abstract

Dependent effect size estimates are a common problem in meta-analysis. Recently, a robust variance estimation method was introduced that can be used whenever effect sizes in a meta-analysis are not independent. This problem arises, for example, when effect sizes are nested or when multiple measures are collected on the same individuals. In this paper, we investigate the robustness of this method in small samples when the effect size of interest is the risk difference, log risk ratio, or log odds ratio. This simulation study examines the accuracy of 95% confidence intervals constructed using the robust variance estimator across a large variety of parameter values. We report results for both estimations of the mean effect (intercept) and of a slope. The results indicate that the robust variance estimator performs well even when the number of studies is as small as 10, although coverage is generally less than nominal in the slope estimation case. Throughout, an example based on a meta-analysis of cognitive behavior therapy is used for motivation. Copyright © 2013 John Wiley & Sons, Ltd.

Robust variance estimation with dependent effect sizes: practical considerations including a software tutorial in Stata and spss

“…Because the goal of this paper is to provide practical advice to meta‐analysts who wish to implement RVE, we do not cover the mathematical details of the RVE estimator in detail; for this we refer the reader to either Hedges et al . () or Tipton ()…”

“…To test these small sample properties, Hedges et al . () conducted a simulation study using the standardized mean difference effect size; similarly, Tipton () conducted a simulation study using the log odds ratio, log risk ratio, and risk difference effect sizes. Results from these simulations suggest the following guidelines should be used in practice: When estimating an average effect size, RVE can be used with as few as 10 studies. When estimating a meta‐regression coefficient (a slope), RVE performs best when there are at least 40 studies and, within these studies, when there are on average five effect sizes per study. When estimating a meta‐regression coefficient (a slope) with fewer than 40 studies, RVE tends to produce CIs that are too narrow (i.e., p ‐values are too small).…”

Handling Complex Meta-analytic Data Structures Using Robust Variance Estimates: a Tutorial in R

“…9), adjustment matrices A j are found on either side of the estimator. In the original formulation of RVE found in Hedges et al [22], these adjustments were not included, and the estimator was known to under-estimate the true variance when the number of studies is small (see simulations in [22,23]). Tipton [20] showed that the bias from inclusion of a small number of studies could be reduced if these adjustment matrices were included.…”

“…RVE is a method that can be used to analyze statistically dependent effect sizes in a meta-analysis [22,23]. Although this method does not require researchers to correctly specify the dependence structure of the data, it is helpful for researchers to understand where and how dependence can arise.…”

“…Although more technical presentations of the RVE method exist (e.g. [22,23]), as well as a tutorial for implementing RVE in SPSS and Stata [24], this is the first tutorial of which we are aware that provides a user-friendly description of how to implement RVE in the R environment.…”