Single group, pre- and post-test research designs: Some methodological concerns

Abstract

This article provides two illustrations of some of the factors that can influence findings from pre-and post-test research designs in evaluation studies, including regression to the mean (RTM), maturation, history and test effects. The first illustration involves a re-analysis of data from a study by Marsden (2004), in which pre-test scores are plotted against gain scores to demonstrate RTM effects. The second illustration is a methodological review of single group, pre-and post-test research designs (pre-experiments) that evaluate causal relationships between intervention and outcome. Re-analysis of Marsden's prior data shows that learners with higher baseline scores consistently made smaller gains than those with lower baseline scores, demonstrating that RTM is clearly observable in single group, pre-post test designs. Our review found that 13% of the sample of 490 articles were evaluation studies. Of these evaluation studies, about half used an experimental design. However, a quarter used a single group, pre-post test design, and researchers using these designs did not mention possible RTM effects in their explanations, although other explanatory factors were mentioned. We conclude by describing how using experimental or quasi-experimental designs would have enabled researchers to explain their findings more accurately, and to draw more useful implications for pedagogy.

Foreign Language Educators’ Exposure to Research: Reported Experiences, Exposure Via Citations, and a Proposal for Action

“…Our findings, from both parts of the study, remind researchers not to ‘drop the ball’ of communicating our research. The current “methodological turn” (Byrnes, , p. 825) in our field has the long‐term ambition of increasing usefulness by improving the insights that research can offer practice (e.g., synthesising findings; replicating; broadening our participant demographics; strengthening study designs; and improving methodological transparency, instrument reliability, and statistical reporting; see, e.g., Marsden, Mackey, & Plonsky, ; Marsden et al., in press; Marsden & Torgerson, ; Ortega, ; Plonsky, ; Plonsky & Derrick, ; Plonsky & Oswald, ). However, systematic, sustained effort in the research–practice interface is needed to make this effort worthwhile, inasmuch as relevance and usefulness were in fact much less of a concern than time and physical and conceptual access, for our respondents at least.…”

Analysis of network interventions with an application to hospital‐acquired infections

“…Third, the treatment assignation procedure as described was based on a known static network; while it is beyond the scope of this work, it would be very worthwhile exploring the effects, especially on power, of needing to find the optimal treatment assignation u based on A 0 or an estimate of the unknown A 1 . Finally, there may be confounding factors in a pre‐post study design that need to be addressed such as contemporaneous effects of innovations in practice and policy that may account for some or all of the observed change . While it is important to continue to study and resolve these issues, this work provides a starting point for developing study designs that investigate the effect of an intervention applied to a subset of network actors on the network as a whole.…”

“…Finally, there may be confounding factors in a pre-post study design that need to be addressed such as contemporaneous effects of innovations in practice and policy that may account for some or all of the observed change. 65 While it is important to continue to study and resolve these issues, this work provides a starting point for developing study designs that investigate the effect of an intervention applied to a subset of network actors on the network as a whole.…”

Engagement in the Overdose RIsk InfOrmatioN (ORION) e-Health Tool for Opioid Overdose Prevention and Self-Efficacy: A Preliminary Study

“…This limitation is not unusual in e-Health interventions 28 . Nonetheless, pre-post design possible flaws were sufficiently addressed since it is unlikely that participants can experience different contemporaneous effects of other informative sources (history) or can mature during such a limited amount of time (maturation) 29 . However, we cannot exclude test effects, attributable to factors such as participants remembering questions or the questions raising awareness and triggering learning after the pre-test.…”