The Use of Prospective Versus Retrospective Pretests with Child-Welfare Involved Families

Abstract

The purpose of the current study is to examine the use of both a prospective and a retrospective pretest in evaluating the impact of the Strengthening Families Program on foster care involved families affected by parental substance abuse. Debate has existed in the literature for over 60 years regarding the use of retrospective pretests in assessing self-reported program impact, with key stakeholders often arguing strongly that, due to response-shift bias, sensitization, and the nature of many human service settings, retrospective pretesting may be more appropriate. However, program evaluators must also incorporate into the evaluation design funding mandates to collect data at specific points in time. In order to mitigate potential pretest-only biases and enhance the evaluation's rigor, this study sought to address the pretest debate through the use of both a traditional, prospective pretest and a retrospective pretest. Using data provided by 411 caregivers, program effectiveness was measured in the areas of family, child and parent functioning. Statistical significance tests and effect sizes were analyzed to compare traditional prospective pretest to posttest scores and retrospective pretests to posttest scores. Findings indicate that overall, the Strengthening Families Program positively impacted family, child and parent functioning and that there were few differences between testing approaches when testing for statistical significance; however, relative differences between prospective pretests and retrospective pretests appeared more prominent in effect size computations. This research informs the longstanding debate, and suggests that program evaluators consider the advantages and disadvantages of using a retrospective pretest in design planning.

Strategies to Avoid Replication Failure With Evidence-Based Prevention Interventions: Case Examples From the Strengthening Families Program

“…Following exposure to self-monitoring assignments and parenting skills activities, parents are more aware of their deficiencies and rate themselves lower. Brook, Akin, Lloyd, Bhattarai, and McDonald (2016) provided evidence that the retrospective pretest is more accurate and matched the implementers’ ratings of the families. The retrospective pretest–posttest design can provide more “veridical” assessments of self-behavior, owing to giving participants realistic anchors and avoiding response shift bias (Chang & Little, 2018).…”

“…As Strobe and Strack (2014) pointed out, “Even multiple failures to replicate an established program finding would not result in a rejection of the original hypothesis, if there are also multiple studies that supported that hypothesis” (p. 64). Since there are multiple studies supporting SFP 10–14 effectiveness, there is sufficient evidence supporting favorable SFP outcomes obtained from RCTs with the 14-session SFP (Kumpfer et al, 2002; Puffer et al, 2017), evaluations relying on propensity matching techniques (Brook et al, 2016) and numerous quasi-experimental, large sample field studies (Kumpfer et al, 2010, 2012).…”

The Online Adaptation and Outcomes of a Family-Based Intervention Addressing Substance Use Disorders

“…However, there still exist reliability concerns with either method—effects of the aforementioned lack of trust at the beginning of the classes (and possibly social desirability bias), and potential memory bias, social desirability bias, or acquiescent response set bias with the retrospective responses. A more recent study of Strengthening Families comparing prospective to retrospective pretest approaches found that although both approaches were similar in terms of statistically significant change, the effect sizes of change scores from the retrospective approach were higher (Brook et al, 2016).…”

Bridging the Gap Between Research and Policy: Fostering Advocacy and Policy Engagement in Special Education Doctoral Students

“…Retrospective pretesting was used because previous research on survey response validity suggests respondents tend to inflate self-ratings prior to a learning experience when they are not able to self-evaluate competencies accurately given the lack of exposure to a topic (Nimon, Zigarmi, & Allen, 2011). While prospective pretest methods follow traditional experimental design methodology, participants in this intimate PD model (i.e., working closely with the instructors for several days in a small cohort) may be more likely to be affected by social desirability bias and specifically worried about exposing their shortcomings prior to building relationships (e.g., Brook, Akin, Lloyd, Bhattarai, & McDonald, 2016). Therefore, the hope was retrospective pretesting would allow doctoral students a chance to become comfortable before being asked for an honest critique and analysis of their experiences.…”