Using response ratios for meta-analyzing single-case designs with behavioral outcomes

Abstract

Methods for meta-analyzing single-case designs (SCDs) are needed to inform evidence-based practice in clinical and school settings and to draw broader and more defensible generalizations in areas where SCDs comprise a large part of the research base. The most widely used outcomes in single-case research are measures of behavior collected using systematic direct observation, which typically take the form of rates or proportions. For studies that use such measures, one simple and intuitive way to quantify effect sizes is in terms of proportionate change from baseline, using an effect size known as the log response ratio. This paper describes methods for estimating log response ratios and combining the estimates using meta-analysis. The methods are based on a simple model for comparing two phases, where the level of the outcome is stable within each phase and the repeated outcome measurements are independent. Although auto-correlation will lead to biased estimates of the sampling variance of the effect size, meta-analysis of response ratios can be conducted with robust variance estimation procedures that remain valid even when sampling variance estimates are biased. The methods are demonstrated using data from a recent meta-analysis on group contingency interventions for student problem behavior.

Evaluation of publication bias in response interruption and redirection: A meta‐analysis

“…Three separate ES indices were calculated for published and unpublished studies. The three ES indices were PND, LLR, and Hedges' g. LRR was transformed and is presented as the percentage of behavior change between baseline and intervention (Pustejovsky, 2018). PND, LRR, and Hedges' g results are presented in Figure 2.…”

“…We then calculated the percentage change defined as: %Δ = 100% x [exp (Ψ) ‐ 1]. Pustejovsky (2018) suggests this transformation when reporting LRR ESs to offer an intuitive way to interpret the magnitude of the effect.…”

“…Log Response Ratio (LRR) is a mean‐based ES index that allows for quantifying the magnitude of behavior change between two adjacent conditions (e.g., baseline and intervention). The LRR is calculated using a natural logarithm transformation that allows for effects to be quantified using the percentage of change between conditions (Pustejovsky, 2018). For example, in a hypothetical experiment to evaluate effects of an intervention to reduce challenging behavior, a LRR of 76.6 represents a 76.6% reduction of the behavior from baseline to intervention.…”

“…For example, in a hypothetical experiment to evaluate effects of an intervention to reduce challenging behavior, a LRR of 76.6 represents a 76.6% reduction of the behavior from baseline to intervention. The LRR metric has three distinct advantages: (a) LRR is not sensitive to procedural variations, such as varying duration of the experimental sessions; (b) the magnitude of the LRR values is not impacted by differences in behavioral measurement systems across studies (e.g., frequency versus interval); and (c) LRR can be used to synthesize outcomes across studies that use different measurement systems (Pustejovsky, 2018). Hedge's g, another mean‐based ES metric for SCD, is a variation of Cohen's d (Cohen, 1988), commonly used in meta‐analyses with group‐design research (e.g., Deci et al, 1999).…”

Confidence Intervals for Ratios of Means and Medians

“…As explained by Pustejovsky (2018), a percent relative difference can be defined as 100% ( μ 2 − μ 1 ) /μ 1 = 100% (ϕ − 1 ) or 100% ( τ 2 − τ 1 ) / τ 1 = 100% (θ − 1 ) and is another unitless measure of effect size that has a useful and simple interpretation. The recommended confidence intervals for ϕ and θ in either an independent-samples design or a paired-samples design can be used to obtain a confidence interval for a percent relative difference by simply replacing ϕ or θ with their lower and upper limits in 100% (ϕ − 1 ) or 100% (θ − 1 ) .…”