The retrospective pretest–posttest design redux: On its validity as an alternative to traditional pretest–posttest measurement

Abstract

We revisit the merits of the retrospective pretest–posttest (RPP) design for repeated-measures research. The underutilized RPP method asks respondents to rate survey items twice during the same posttest measurement occasion from two specific frames of reference: “now” and “then.” Individuals first report their current attitudes or beliefs following a given intervention, and next they are prompted to think back to a specific time prior to the given intervention and rate the item again retrospectively. The design addresses many of the validity concerns that plague the traditional pretest–posttest design. Particularly when measuring noncognitive constructs, the RPP design allows participants to gauge the degree of change that they experience with greater awareness and precision than a traditional approach. We review the undesirable features of traditional designs and highlight the benefits of the retrospective approach. We offer examples from two recent, original studies and conclude with the recommendation that the RPP design be employed more broadly. We also conclude with a discussion of important directions for future examination of this design.

The impact of COVID-19 lockdown on social network sites use, body image disturbances and self-esteem among adolescents and young women

“…Third, causality cannot be assumed among variables as the study had a cross-sectional design. Even though the retrospective design has been found to be a valuable alternative to traditional pretest-posttest design as it addresses many of its validity concerns and mitigates the response shift bias that seems ubiquitous in the traditional approach, criticisms of the retrospective design include memory-related problems, impression management and response bias (Little et al 2020).…”

From quality to outcomes: a national study of afterschool STEM programming

“…The selection of a youth self-report survey took into consideration existing peer-reviewed literature as well as input from state networks and practitioners regarding outcomes that align with current programmatic goals and outcomes that are relevant across different types of STEM programming. With this in mind, the survey chosen assessed five STEM attitudes (i.e., STEM engagement, career interest and knowledge, activity participant, and identity) and four SEL/twenty-first-century skills (i.e., perseverance, critical thinking, relationships with adults, and relationships with peers) that are associated with success in STEM, both academically and professionally-and in this case, the survey was the Common Instrument Suite for Students (CIS-S, Little et al, 2019;Noam, Allen, Shah, & Triggs, 2017;Sneider & Noam, 2019).…”

“…Conceptualization of CIS-S scales STEM engagement. This construct was measured using the Common Instrument, a validated self-report survey of STEM engagement that was developed in partnership with researchers and practitioners in the informal STEM education field (Little et al, 2019;Martinez et al, 2014;Noam et al, 2017;Sneider & Noam, 2019). This scale captures three aspects of engagement: behavioral (e.g., participating or involving oneself in STEM activities or projects), cognitive (e.g., to be drawn to understanding, observing, or figuring out STEM phenomena), and "I like to think of different ways to solve a problem.…”

“…Administration took approximately 15 min. All attitudinal items on the student assessment were in retrospective pretest-posttest format (Little et al, 2019), and students' responses were recorded using a visual analog scale (VAS), a continuous scale of measurement. Each scale ranged from 0 (strongly disagree) to 99 (strongly agree), with a score of 49 representing the midpoint (neutral).…”

“…Third, we acknowledge the potential concerns related to the use of a retrospective pretest-posttest design to measure change in students' attitudes and beliefs, including memory-related problems (e.g., memory distortion, selective perception, and poor memory, especially among children and adolescents), social desirability, and impression management and response bias (for review, see Little et al, 2019). Despite these concerns, we determined that the advantages of using a retrospective survey design outweighed the disadvantages in this particular case.…”