Identification of Real and Artifactual Moderators of Effect Size in Meta-Analysis

Abstract

This article argues that while meta-analytic studies are widely used in psychological literature, heterogeneity and the potential for confounding remain major problems in the interpretation of meta-analytic study results. The article demonstrates the use of exploratory analysis including graphical methods prior to meta-analysis, and introduces a methodology to screen for artifactual effects. These procedures are illustrated on effect size data comparing depression treatment outcome from psychotherapy versus pharmacotherapy. Results support prior findings of a nonsignificant difference in effect size between the two treatments. They also support findings that treatment type accounts for only a very small proportion of outcome variance. However, the results indicate that some previously reported covariates of depression treatment outcome may be artifactual.

Improving professional psychological practice through an increased repertoire of research methodologies: Illustrated by the development of MOL.

“…While causal conclusions cannot be drawn from these associations, it does appear to be the case that, in the Figure 1. Efficiency ratios calculated on the studies identified by Collins and Carey (2015) and averaged across year of publication. This document is copyrighted by the American Psychological Association or one of its allied publishers.…”

“…The efficiency of treatments is also decreasing over time. Using the studies from Collins and Carey (2015), efficiency ratios (Carey, Tai, & Stiles, 2013) were conducted on all studies that provided effect size and number of sessions data. These ratios were averaged for studies that were published in the same year.…”

“…RCTs have not served us well in the development of more effective and efficient psychological treatments. Two separate and independent meta analyses (Collins & Carey, 2015; Johnsen & Friborg, 2015) reported a linear decline in pre-post effect sizes for the psychological treatment of depression from before 1980 until 1995 when effect sizes appeared to stabilize. Given that the purpose of RCTs is to provide evidence to improve clinical practice (Salter & Louise, 2015), it is hard to understand a large increase in RCTs (Carey & Stiles, 2015) and a reduction in the effect sizes of therapies being evaluated.…”

“…Efficiency ratios calculated on the studies identified by Collins and Carey (2015) and averaged across year of publication.…”