The Power of Teaching Activities: Statistical and Methodological Recommendations

Abstract

Researchers rarely mention statistical power in Teaching ofPsychology teaching activity studies. Insufficiently powered tests promote uncertainty in the decision to accept or reject the tested null hypothesis and influence the interpretation of results. We analyzed the a priori power of statistical tests from 197 teaching activity effectiveness studies published from 1974 through 2006. We found that two thirds of the studies were powerful enough to detect only large effects. We compared observed sample sizes with expert recommendations and found that studies typically used sample sizes that were too small. We discuss limitations of underpowered statistical tests in evaluating teaching activity effectiveness and make design-related recommendations for improving power.

Meta-Analysis of Group Learning Activities: Empirically Based Teaching Recommendations

“…To accomplish these goals, we conducted the following meta-analysis of collaborative learning effect sizes. Given that effect sizes are determined in part by power and teaching research suffers from low power (Tomcho & Foels, 2009), we would expect this limited power to impact the potential ability to detect relationships in our meta-analysis (see Hedges & Pigott, 2001). Additionally, notable researchers have argued that adopting a more liberal Type-I error rate is advisable in exploratory research (see Cohen, 1992; Hallahan & Rosenthal, 1996).…”

“…Additionally, notable researchers have argued that adopting a more liberal Type-I error rate is advisable in exploratory research (see Cohen, 1992; Hallahan & Rosenthal, 1996). In the interests of maximizing the information that could be gleaned from the database, we adopted a more liberal Type-I error rate of α = .10 (see Cohen, 1990; Rosnow & Rosenthal, 1989; Tomcho & Foels, 2009) for our hypothesis testing to allow for explorations of moderators that might offer potential applicability in the translation of research into classroom practice. Thus, we sought to identify potential factors that warrant both additional study and also offer applicability in the classroom.…”

Searching for Significance in the Scholarship of Teaching and Learning and Finding None: Understanding Non-Significant Results

“…This is because statistical power increases as sample size increases. Recently, Tomcho and Foels (2009) reviewed the power of statistical tests from 197 studies published in Teaching of Psychology. Their findings provide an important reminder about interpreting non-significant effects.…”

“…Prior to conducting a study, researchers should plan for the appropriate sample size to achieve 80% power (statistical power calculators can be found online). Specific recommendations on ways to increase power are provided by Tomcho and Foels (2009) and readers may also wish to consult Cohen's (1988) work on the topic. Lastly, Tomcho and Foels (2009) recommend that researchers report statistical power to aid the reader's interpretation of results, and I believe this is an important consideration to further the work of our field.…”

“…Specific recommendations on ways to increase power are provided by Tomcho and Foels (2009) and readers may also wish to consult Cohen's (1988) work on the topic. Lastly, Tomcho and Foels (2009) recommend that researchers report statistical power to aid the reader's interpretation of results, and I believe this is an important consideration to further the work of our field. Providing more statistical information in our articles (e.g., power, effect size, CIs) and being cognizant of what findings from NHST can and cannot tell us will lead to more robust conversations and investigations about teaching and learning.…”