A follow‐up power analysis of the statistical tests used in the Journal of Research in Science Teaching

Abstract

It has been two decades since the first power analysis of a psychological journal and 10 years since the Journal of Research in Science Teaching made its contribution to this debate. One purpose of this article is to investigate what power-related changes, if any, have occurred in science education research over the past decade as a result of the earlier survey. In addition, previous recommendations are expanded and expounded upon within the context of more recent work in this area. The absence of any consistent mode of presenting statistical results, as well as little change with regard to power-related issues are reported. Guidelines for reporting the minimal amount of information demanded for clear and independent evaluation of research results by readers are also proposed.

The effect of writing assignments on achievement in college general chemistry

“…This means that the significance tests employed had a 63% chance of detecting significant departures from the null hypothesis as large as 0.5 standard deviation units. Although a larger value for power is desirable, the obtained value of 0.63 is identical with the average power reported in a recent power analysis of 192 studies reported in JRST (Woolley and Dawson, 1983). To get a further indication of the power of the statistical tests, the obtained effect size for the study was computed using the pooled standard deviation for both samples (14.9) and the adjusted difference between sample means (7.8 points).…”

Effect Size Use in Studies of Learning Disabilities

“…Furthermore, published concerns about the misinterpretation of significance testing and published efforts to advocate alternatives have had little influence on actual practice (B. Thompson, 1997;Woolley & Dawson, 1983;Young, 1993). Several reviews of journals in these fields have confirmed this resistance to change in the use of signifi-cance testing (Dar, Serlin, & Omer, 1994;Kirk, 1996;Plucker, 1997;B.…”

The Power of Teaching Activities: Statistical and Methodological Recommendations

“…Relative to findings in the field of science teaching, ToP researchers were as likely to detect large or medium effects as were other researchers (e.g., Woolley & Dawson, 1983). Given Osborne et al's (2001) and findings that psychology teaching research results in a medium effect, ToP researchers appear just as likely to be able to detect effects as researchers teaching in other areas of science.…”

“…Comparable to findings in the field of psychology (e.g., Cohen, 1962;Rossi, 1990;Sedlmeier & Gigerenzer, 1989), ToP researchers more likely detected large than medium or small effects. In a secondary analysis, ToP researchers' power to detect effects was generally similar to teaching-related journals in other disciplines (e.g., Journal of Research in Science Teaching [JRST]) for medium and large effect sizes, but less for small effect sizes (e.g., Daly & Hexamer, 1983;Penick & Brewer, 1972;Woolley, 1983;Woolley & Dawson, 1983). For example, Woolley and Dawson (1983) used a two-tailed alpha of .05 for each hypothesis test in examining 192 articles published in JRST and found statistical power estimates of .23, .63, and .85, for small, medium, and large effects, respectively.…”

“…In a secondary analysis, ToP researchers' power to detect effects was generally similar to teaching-related journals in other disciplines (e.g., Journal of Research in Science Teaching [JRST]) for medium and large effect sizes, but less for small effect sizes (e.g., Daly & Hexamer, 1983;Penick & Brewer, 1972;Woolley, 1983;Woolley & Dawson, 1983). For example, Woolley and Dawson (1983) used a two-tailed alpha of .05 for each hypothesis test in examining 192 articles published in JRST and found statistical power estimates of .23, .63, and .85, for small, medium, and large effects, respectively. We recalculated statistical power for each hypothesis test in our sample using a two-tailed alpha equal to .05 for each test (rather than controlling for experiment-wise error rates) to compare our results with Woolley and Dawson.…”