A Complete SMOCkery: Daily Online Testing Did Not Boost College Performance

Abstract

In an article published in an open-access journal, (Pennebaker et al. PLoS One, 8(11), e79774, 2013) reported that an innovative computer-based system that included daily online testing resulted in better student performance in other concurrent courses and a reduction in achievement gaps between lower and upper middle-class students. This article has had high impact, not only in terms of citations, but it also launched a multimillion-dollar university project and numerous synchronous massive online courses (SMOCs). In this study, I present a closer look at the data used in the Pennebaker et al. study. As in many cases of false claims, threats to internal validity were not adequately addressed. Student performance increases in other courses can be explained entirely by selection bias, whereas achievement gap reductions may be explained by differential attrition. It is hoped that the findings reported in this paper will inform future decisions regarding SMOC courses. More importantly, our field needs watchdogs who expose such unsupported extravagant claims—especially those appearing in pay-to-publish journals.

Wild brooms and learning analytics

“…If a high-performing subset of students use an analytical service more than the average student, correlational statistics would fallaciously imply that the service causes improvements in performance, and this correlation would be even stronger if occasional use harms average or low-performing students. Similarly, if an analytical service causes students to withdraw from a course or an institution, and withdrawing also removes these students from analysis, paradoxically this harmful service may appear to improve student performance when benchmarked against comparison groups (attrition bias; e.g., Robinson 2021 ), as may have been the case with the traffic signal example above (Straumsheim, 2013 ). In this way, we imagine institutional analytics initiatives not only incanting dangerous spells, but potentially doing so blindfolded, or worse: under the mistaken impression that the spell is working effectively.…”

Wild Brooms and Learning Analytics

“…If a high-performing subset of students use an analytical service more than the average student, correlational statistics would fallaciously imply that the service causes improvements in performance, and this correlation would be even stronger if occasional use harms average or low-performing students. Similarly, if an analytical service causes students to withdraw from a course or an institution, and withdrawing also removes these students from analysis, paradoxically this harmful service may appear to improve student performance when benchmarked against comparison groups (attrition bias; e.g., Robinson, 2021), as may have been the case with the traffic signal example above (Straumsheim, 2013). In this way, we imagine institutional analytics initiatives not only incanting dangerous spells, but potentially doing so blindfolded, or worse: under the mistaken impression that the spell is working effectively.…”