Magnitudes of Experimental Effects in Social Science Research

Abstract

How the magnitude of an experimental effect may be measured has been a matter of concern for at least two decades. The phenomenon of effect size is still not well understood, and it cannot be inferred from statistical significance. In recent years various ways of assessing the amount of variance accounted for have been proposed as measures of magnitude of effect. Other writers have proposed rulesfor standardizing effect size, with the interpretations of the measures depending largely on intuitions buttressed by some further general empirical norms. All the methods of assessing effect size have serious flaws that limit their usefulness. The various statistical procedures for estimating variance accounted for are based on different statistical models and can produce rather sharply differing results, depending on the model employed. All the methods suffer from the limitation that they reflect to too great an extent the particular characteristics of the study being reported and hence have limited generalizability. problem of which researching psychologists have been aware for years (e.g., Bolles and Messick, 1958;Savage, 1957) but that has had increasing attention over the past decade or so is how to determine how large an effect is achieved by an experimental intervention. Especially for psychologists working in applied areas it is important to know more than that a treatment produces a statistically significant main effect. However, even for theoretical problems it is at least enlightening and often sobering to find out how much an effect is at stake during the intricacies of a theoretical controversy.

Measures of Effect Size for Comparative Studies: Applications, Interpretations, and Limitations

“…But several critics have pointed out that measures of association are affected by the reliability of the measures, the heterogeneity of the populations being compared, the specific levels of the variables studied, the strength of the treatments, and the range of treatments (Maxwell et al, 1981;O'Grady, 1982;Sechrest & Yeaton, 1982), thus making such comparisons hazardous. Fern and Monroe (1996) presented an especially comprehensive discussion of these factors.…”

“…The strength of the treatment refers to the likelihood that the treatment will have the intended effect (Sechrest & Yeaton, 1982). A strong treatment would explain more variance than a weak treatment but the quantification of the strength of a treatment is generally unknown.…”

“…The amount of variation in the word knowledge scores explained by both of these factors depended a great deal on the type of cues provided in the context and the clarity of the definitions provided. Sechrest and Yeaton (1982) argue that without knowledge of the strength of a treatment the proportion of variance accounted for is meaningless.…”

“…Over the years, several papers have been published tracing the historical developments of effect size measures (Dwyer, 1974;Glass & Hakstian, 1969;Maxwell, Camp, & Arvey, 1981;Richardson, 1996), suggesting alternative indices (Friedman, 1968;McGraw & Wong, 1992;Rosenthal & Rubin, 1982), and offering cautionary notes on the interpretation of these indices (Mitchell & Hartmann, 1981;Muray & Dosser, 1987;O'Grady, 1982;Sechrest & Yeaton, 1982;Strube, 1988). Textbooks on statistical methods have also included procedures for computing a number of indices of effect (e.g., Huberty, 1994;Keppel, 1991;Maxwell & Delaney, 1990;Stevens, 1996).…”

Evaluation Practices of Educational Programs Reviewed By the Joint Dissemination Review Panel, 1980-1983

“…Recall that the magnitude of effect size is dependent on two factors: the difference between the treatment and comparison groups and the amount of variance that exists within the study. To the extent that the evaluator can reduce extraneous variance through increased precision of measuring instruments, or through careful planning and implementing of the evaluation design, the size of the effects detected will increase, other things being equal (Hall, 1980;Sechrest and Yeaton, 1982). While these features are certainly desirable for all evaluation research, when they do not exist consistently across a sample of educational programs being compared, the interpretation of effect size for any given program is confounded with the quality of the evaluation design.…”