The power of statistical tests in science teachnig research

Abstract

A calculation of the probability of rejecting H0 when it should be rejected (power) was completed on each of the 66 applicable articles in Volumes 6 and 7 (1969, 1970) of the Journal of Research in Science Teaching. These power calculations utilized the effect size definitions and tables developed by Cohen (1969). The mean power of each article to detect small, medium, and large effect sizes was determined from its major statistical tests. These mean powers were then compiled and analyzed. The powers calculated for the different effect sizes were disturbingly low (small, 0.22; medium, 0.71; large, 0.87) but not generally as low as Cohen (1962) found in an analysis of another behavioral journal. Recommendations for improving confidence in research in science teaching is provided and centers on significant increases in sample sizes and an understanding of power and its relation to a, effect size and sample size.

A follow‐up power analysis of the statistical tests used in the Journal of Research in Science Teaching

“…This reflects the all too common practice of testing large matrices of coefficients for all possible pairings of study variables (and inviting the wrath of capitalization on chance). Despite this, however, the relative frequency of each test seems to be in line with what has been seen in other surveys (Chase & Chase, 1976), as well as the earlier analysis of this journal (Penick & Brewer, 1972). Also, the mean sample size for the studies in Volumes 14-17, x = 154.6, has remained relatively fved in JRST over the past decade (see Table 11).…”

“…Since that time the statistical power of published tests has been at issue in fields as diverse as communication (Chase & Tucker, 1975;Katzer & Sodt, 1973), gerontology (Levenson, 1980), and medicine (Freiman et al, 1978). Ten years have passed since the Journal of Research in Science Teaching made its contribution to this debate through the work of Penick and Brewer (1972).…”

“…One purpose of this article is to investigate what power-related changes, if any, have occurred in science education research over the past decade as a result of the survey by Penick and Brewer (1972) and the multitude of other researchers who subsequently wrote on this topic. In addition, it is hoped that the recommendations of Penick and Brewer can be expanded and expounded upon within the context of more recent work in this area.…”

Conducting tests of hypotheses: The need for an adequate sample size

“…However, if they cannot tolerate the adjusted levels of alpha, power or both, that is, if alpha is too high or power too low to be justifiable from a research point of view, then they should consider postponing the study until they can get a larger sample. Cohen (1962), Brewer (1972), Jones and Brewer (1972), Penick and Brewer (1972), Brewer and Owen (1973), Katzer and Sodt (1973), and Chase and Chase (1976), respectively, the researchers found a dismal state of affairs with regard to the power of the tests conducted, as indicated in Table 2. Table 2: Results of studies on the power of hypotheses tests…”

“…With the exception of the studies analysed by Penick and Brewer (1972) and Brewer and Owen (1973), in which the mean power of the tests was computed at .71 and .72 respectively, the rest of the tests in the journals surveyed had a more or less 50-50 chance of detecting a medium effect. This implies that the researchers who conducted hypothesis testing at such low levels of power would have saved a lot of time and energy had they just tossed a coin in deciding whether or not to reject since the probability of a correct rejection was approximately 1 /2.…”