Trials and tribulations in the meta-analysis of treatment differences: Comment on Wampold et al. (1997).

Abstract

A fair test of the Dodo bird conjecture that different psychotherapies are equally effective would entail separate comparisons of every pair of therapies. A meta-analysis of overall effect size for any particular set of such pairs is only relevant to the Dodo bird conjecture when the mean absolute value of differences is 0. The limitations of the underlying randomized clinical trials and the problem of uncontrolled causal variables make clinically useful treatment differences unlikely to be revealed by such heterogeneous meta-analyses. To enhance implications for practice, the authors recommend an intensified focus on patient-treatment interactions, cost-effectiveness variables, and separate metaanalyses for each pair of treatments. Wampold et al. (1997) examined studies that directly compared "bona fide" treatments [i.e., treatments that "were based on psychological principles, were offered to the psychotherapy community as viable treatments (e.g., through professional books or manuals)," and "were delivered by trained therapists" (p. 205; with at least a master's degree)] to patients with bona fide clinical problems. The results of their analyses are consistent with those of prior meta-analyses, and proponents of psychotherapy can be reassured by the convergence of their findings. For example, Lipsey and Wilson (1993) examined 156 meta-analyses in which treatments were compared with control conditions. They calculated a mean effect size of .47, which was considerably larger than the mean effect size of many widely used, "validated" medical interventions. Grissom (1996) calculated "probability of superiority estimates" (cf. Howard, Krause, & Vessey, 1994) from prior meta-analyses. His analysis indicated that, in general, therapy was much better than no treatment and better than a placebo and that the median probability of superiority for studies comparing two therapies was only slightly greater than 50-50. So Wampold et al.'s meta-analysis is in a tradition of results indicating that efficacy differences between psychotherapeutic treatments are, on the average, modest to small. How to Compare TreatmentsIf we look for sheer differences in outcome among psychotherapies to see whether they are all the same in terms of the

Psychodynamic Measures of Therapeutic Change

“…This frequently observed "dodo bird verdict" (e.g., Luborsky, 1975;Luborsky et al, 1995;Wampold, Mondin, Moody, Stich, et al, 1997; but see objections by Crits-Cristoph, 1997, andHoward et al, 1997, and a rejoinder by Wampold, Mondin, Moody, and Ahn, 1997), so named because of the dodo bird's statement in Alice in Wonderland that all have won and all must have prizes, is suggested by extensive meta-analyses that indicate few differences between alternative forms of treatment (e.g., Frank, 1979;Smith, Glass, and Miller, 1980;American Psychiatric Association Commission on Psychotherapies, 1982;Shapiro and Shapiro, 1982). These similarities suggest that either common processes are shared by various therapeutic approaches (Frank, 1982;Strupp and Binder, 1984), such that they are functionally equivalent (Lambert, Shapiro, and Bergin, 1986;Stiles, Shapiro, and Elliot, 1986), or that research has not adequately addressed major methodological issues (e.g., VandenBos and Pino, 1980;Wortman, 1983;Kazdin, 1986).…”

Let's face facts: Common factors are more potent than specific therapy ingredients.

“…Despite this record going back almost 25 years to the meta-analysis of therapy outcomes performed by Smith and Glass (1977), Luborsky et al (this issue), Crits-Christoph (1997), and others continue to hold out the possibility or even the likelihood of finding significant differences among the therapies. In a similar vein, Howard, Krause, Saunders, and Kopta (1997) suggested that therapies should be ordered along an efficacy continuum. Luborsky et al's meta-meta-analysis places us no closer to either goal than have the previous meta-analyses, precisely because the evidence points to all active therapies being equally beneficial.…”

“Between-group psychotherapy outcome research and basic science” revisited

“…effect of any therapy, only, at best, its difference from the others in terms of patient outcome. One therapy may be much better than another and good enough in one randomized experiment without being good enough at the other levels of the unmanipulated variables that we would encounter in another such experiment (Howard, Krause, Saunders, & Kopta, 1997). Thus, we cannot conclude from a statistically proper interpretation of the results of randomized clinical trials if any of the therapies compared should be assigned to any specific patient even though we have evidence for which is the best on average at the unique (but undefined) location or, more likely, set of locations in C at which the trials occurred.…”