Difference-in-Differences as an Alternative to Pretest–Posttest Regression for Social Work Intervention Evaluation and Research

Abstract

Nonrandomized evaluation designs are an important part of social work research because randomization is not always feasible in social work settings. Although randomly assigned groups are assumed to be equivalent, nonrandomly assigned groups are not. In nonrandomized settings, designs with multiple waves are ideal, but two-wave designs are still widely used. A common method for estimating a treatment effect in nonrandom two-wave designs is the pretest–posttest model. However, depending on relationships among participants and the method of assignment to treatment groups, researchers should consider a difference-in-differences approach to testing treatment effects. Authors describe and compare the pretest–posttest and difference-in-differences approaches and assumptions and offer guidelines, developed from a literature review, about the conditions under which each model is likely to be best. Authors also demonstrate the decision-making process and application of the methods in an evaluation of an elementary school intervention program.

Evaluating a Functional Family Therapy Intervention: Challenges and Lessons Learned

“…The practical implications of randomized trials are often limited. For example, complex interventions often preclude a clear understanding of which aspects of the intervention were helpful and which were not (Butts & Roman, 2018; Chemers & Reed, 2005; Rose & Bowen, 2019). According to some scholars, conducting randomized clinical trials among culturally diverse clients is especially challenging (Marsiglia & Booth, 2013; Sampson & Torres, 2015).…”

From A to Z: Effects of a 2nd-grade reading intervention program for struggling readers

“…It is important to stress that random assignment of students was impossible because many teachers, although identifying reading difficulties in some children with results < 30 wpm in the oral reading test, argued that children with more difficulties other than these should be assigned to the program. The difference-in-differences design of our study, however, is a tool that minimizes the effect of the non-randomization of participants (Rose & Bowen, 2019;Steinmann et al, 2023;Strello et al, 2021).…”

Positive Adolescent Development: Effects of a Psychosocial Intervention Program in a Rural Setting

“…This is the recommended analysis when working with non-random or pre-existing groups [ 57 ], as in the case when schools are assigned to conditions [ 58 ].The eta-square partial ( ) value is used as an indicator of the effect size. Cohen [ 59 ] suggested that ≤ 0.06 can be considered a ‘small’ effect size, 0.07 ≤ ≤ 0.14 represents a ‘medium’ effect size, and > 0.14 is a ‘large’ effect size.…”