Statistical power of tests of the situational specificity hypothesis in validity generalization studies: A cautionary note.

Osburn, H. G.; Callender, John C.; Greener, Jack M.; Ashworth, Steven D.

doi:10.1037/0021-9010.68.1.115

Cited by 52 publications

(64 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This can be solved only by doing more primary studies. The good news is that power may be less of a problem in meta-analysis than in primary research (Osburn, Callender, Greener, & Ashworth, 1983;Sackett, Harris, & Orr, 1986;Spector & Levine, 1987). The reason is that the unit of analysis in primary research is usually an individual, but the unit in meta-analysis is a study-level effect size that is an aggregate of these primary units.…”

Section: Problems With Analyzing Moderator Effectsmentioning

confidence: 99%

Mediators and moderators in meta-analysis: There's a reason we don't let dodo birds tell us which psychotherapies should have prizes.

Shadish¹,

Sweeney²

1991

Journal of Consulting and Clinical Psychology

224

164

View full text Add to dashboard Cite

In primary studies, psychotherapy researchers frequently search for mediator and moderator variables that can help them understand the relationship between treatment and outcome. Yet a review of past psychotherapy meta-analyses revealed that none examined the possible role of mediator variables; and although all of them searched for moderators of study outcome, that search was generally not as complete as it could have been. This article illustrates methods for studying such mediator and moderator variables in meta-analysis, discusses their advantages and disadvantages, and shows how the inclusion of these variables can change interpretation of meta-analytic results. In particular, the perennial interpretation of past psychotherapy meta-analyses that therapeutic orientation makes no difference to outcome--or as the dodo bird put it: "Everyone has won and all must have prizes'--may be wrong. Orientation may make significant difference, but only by virtue of its moderating and mediating effects.To the best of our knowledge, all meta-analyses ever done have concluded that (on the average) clients receiving psychotherapy do better than clients not receiving psychotherapy. In fact, the computation of average therapy effects over studies is the defining strength of meta-analysis. But this strength leads to a criticism of meta-analysis: Knowledge of average effects says nothing about when, where, why, and how therapy works. The latter questions concern mediators and moderators of therapy outcome. The present article describes methods for addressing such questions in meta-analysis.Moderators and mediators are third variables that help researchers to understand the relationship between independent and dependent variables (Baron & Kenny, 1986). "A moderator is a qualitative (e.g., sex, race, class) or quantitative (e.g., level of reward) variable that affects the direction and/or strength of the relations between an independent or predictor variable and a dependent or criterion variable" (Baron & Kenny, 1986, p. 1174. Moderators cause statistical interactions. Some moderator variables are categorical. Suppose, for example, that behavioral therapies yielded high effect sizes on behavioral presenting problems but low effect sizes on nonbehavioral presenting problems, with the opposite pattern emerging for nonbehav-

show abstract

Section: Problems With Analyzing Moderator Effectsmentioning

confidence: 99%

Mediators and moderators in meta-analysis: There's a reason we don't let dodo birds tell us which psychotherapies should have prizes.

Shadish¹,

Sweeney²

1991

Journal of Consulting and Clinical Psychology

224

164

View full text Add to dashboard Cite

show abstract

“…In the interest of fairness, we believe that it is important also to evaluate the power of the VG ratio in regard to rejecting the null hypothesis of situational specificity. A recent simulation study by Osburn et al (1983) suggested that the decision rule to reject the situational specificity hypothesis when 2/ 2 >.75 results in too much power in the sense that situational specificity is rejected when low to moderate variance exists among the pi (true validities), given that the n are not large (< 100). We wish to address this point with some logic and simple algebra within the context that sample sizes are not large (e.g., .i 70) and for the critical value of the VG decision rule (i.e., > .75).…”

mentioning

confidence: 99%

A note on validity generalization procedures.

James¹,

Demaree²,

Mulaik³

1986

Journal of Applied Psychology

139

View full text Add to dashboard Cite

generalization procedures are reviewed and found to be subject to the logical fallacy of affirming the consequent. Alternative models may explain variation in validity coefficients as well as the cross-situational consistency model espoused by many users of the validity generalization approach. Moreover, some of the assumptions that form the statistical foundation of validity generalization work are open to question. Use of Fisher 2 transformations in validity generalization analyses removes most of these problems and will usually produce more conservative estimates of the degree to which sampling error may account for variability in correlations.The hypothesis that validities are "situationally specific" in the context of personnel research has been addressed in a number of recent studies and reviews (cf.

show abstract

“…In studies where the variance in true validities remains substantial after removing artifactual variation, the lower limits of the credibility interval is used to justify conclusions about the generalizability of the test' validity across settings and organizations (Osbum, Callender, Greener, & Ashworth, 1983;Pearlman et al, 1980;Schmidt ~z Hunter, 1981;Schmidt, Hunter, & Pearlman, 1982;Schmidt et al, 1979;Schmidt, Pearlman, Hunter, & Hirsh, 1985). As described in Schmidt and Hunter (1977), the credibility interval is to have a Bayesian interpretation as an interval centered at the mean of the posterior distribution of true validities.…”

mentioning

confidence: 98%

Tolerance Intervals: Alternatives to Credibility Intervals in Validity Generalization Research

Millsap

1988

Applied Psychological Measurement

View full text Add to dashboard Cite

In validity generalization research, the estimated mean and variance of the true validity distribution are often used to construct a credibility interval, an interval containing a specified proportion of the true validity distribution. The statistical interpretation of this interval in the literature has varied between Bayesian and classical (frequentist) viewpoints. Credibility intervals are here discussed from the frequentist perspective. These are known as "tolerance intervals" in the statistical literature. Two new methods for constructing a credibility interval are presented. Unlike the current method of constructing the credibility interval, tolerance intervals have known performance characteristics across repeated applications, justifying confidence statements. The new methods may be useful in validity generalization research involving a small or moderate number of validation studies.

show abstract

Statistical power of tests of the situational specificity hypothesis in validity generalization studies: A cautionary note.

Cited by 52 publications

References 11 publications

Mediators and moderators in meta-analysis: There's a reason we don't let dodo birds tell us which psychotherapies should have prizes.

Mediators and moderators in meta-analysis: There's a reason we don't let dodo birds tell us which psychotherapies should have prizes.

A note on validity generalization procedures.

Tolerance Intervals: Alternatives to Credibility Intervals in Validity Generalization Research

Contact Info

Product

Resources

About