Why is impact measurement abandoned in practice? Evidence use in evaluation and contracting for five European Social Impact Bonds

Abstract

Despite broad consensus on the importance of measuring “impact,” the term is not always understood as estimating counterfactual and causal estimates. We examine a type of public sector financing, “Social Impact Bonds,” a scheme where investors front money for public services, with repayment conditional on impact. We examine five cases in four European countries of Social Impact Bonds financing active labor market programs, testing the claim that Social Impact Bonds would move counterfactual causal impact evaluation to the heart of policy. We examine first how evidence was integrated in contracts, second the overall evidence generated and third, given that neither contracts nor evaluations used counterfactual definitions of impact, we explore stakeholders’ perspectives to better understand the reasons why. We find that although most stakeholders wanted the Social Impact Bonds to generate impact estimates, beliefs about public service reform, incentives, and the logic of experimentation led to the acceptance of non-causal definitions.

The impact of social impact bond financing

“…This meant the provider had a larger pool of potential participants to choose from, so more selection was possible. Canceling the RCT also improved measured program success, as one contract target was raw enrollment numbers and because employment targets used flat numbers rather than rates (Hevenstone, Fraser, Hobi, & Geuke, 2023).…”

“…The Swiss government did not pursue further SIBs and did not offer the provider further contracts following our study. In the Dutch case, government used stricter and more rigorous contractual terms, more akin to a principle‐agent model in further SIBs (Hevenstone et al, 2023). Both cases in this study were early SIB contracts in their respective locations, meaning success was important, both with respect to political liabilities and the potential to develop further SIBs.…”

Missing finance in social impact bond research? A bibliometric overview between past and future research

“…In overall terms, the SIB research landscape appears to be dominated by the presence of some recurrent themes – such as those identified in the cluster analysis – which is leading the way. The content analysis led to underlining that the vast majority of the literature dealing with SIBs mainly focuses either on the posterior estimation of the impact generated (which is still a highly debated issue, see for example, Fraser et al, 2022; Hevenstone et al, 2022) or on a mere description of the developed SIBs mainly biased by the limited availability of publicly available data. Future works should consider the idea of looking at SIBs more through the lens of finance, considering them as financial instruments involving private investors and with a certain level of risk.…”