Teen Pregnancy Prevention Evaluation Technical Assistance: A Case Study

Abstract

Purpose: This case study discusses Mathematica’s experience providing large-scale evaluation technical assistance (ETA) to 65 grantees across two cohorts of Teen Pregnancy Prevention (TPP) Program grants. The grantees were required to conduct rigorous evaluations with specific evaluation benchmarks. This case study provides an overview of the TPP grant program, the evaluation requirements, the ETA provider, and other key stakeholders and the ETA provided to the grantees. Finally, it discusses the successes, challenges, and lessons learned from the effort. Conclusion: One important lesson learned is that there are two related evaluation features, strong counterfactuals and insufficient target sample sizes, that funders should attend to prior to selecting awardees because they are not easy to change through ETA. In addition, if focused on particular outcomes (for TPP, the goal was to improve sexual behavior outcomes), the funder should prioritize studies with an opportunity to observe differences in these outcomes across conditions; several TPP grantees served young populations, and sexual behavior outcomes were not observed or were rare, limiting the opportunity to observe impacts. Unless funders are attentive to weaning out evaluations with critical limitations during the funding process, requiring grantees to conduct impact evaluations supported by ETA might unintentionally foster internally valid, yet underpowered studies that show nonsignificant program impacts. The TPP funder was able to overcome some of the limitations of the grantee evaluations by funding additional evidence-building activities, including federally led evaluations and a large meta-analysis of the effort, as part of a broader learning agenda.

ETA Special Issue Editor Essay

“…The second paper (Knab and Cole, 2020) considers ETA provided for another Tiered Evidence program, the Teen Pregnancy Prevention Program (TPP), 2 in the Office of Adolescent Health, now part of the Office of Population Affairs in the Department of Health and Human Services (HHS). Here, the ETA was provided to 65 recipients across two cohorts.…”

“…The goals of this focused TA are two-fold: (1) to help grantees produce high-quality evaluations that further build the evidence base in a particular area and (2) to increase evaluation capacity among grantees and their evaluators. The four articles of this special issue (Gothro et al, 2020;Knab & Cole, 2020;Zaid et al, 2021;Zandniapour & Hyde, 2020) demonstrate ETA's potential and its challenges.…”