Impact evaluation of water infrastructure investments: Methods, challenges and demonstration from a large-scale urban improvement in Jordan

Abstract

Impact evaluation (IE) of large infrastructure presents numerous challenges, and investments in urban piped water and sanitation are no exception. Here we present methods for more systematic assessment of the implications of such interventions, discussing tradeoffs between validity, relevance and practicality that arise from alternative approaches. Then, to more clearly illustrate the many issues that typically arise in such IEs, we draw on an example application in Zarqa, Jordan, where the Millennium Challenge Corporation invested about US$275 million to upgrade and extend piped water and sewer networks, as well as increase the capacity of the country's largest wastewater treatment plant. The theory of change for the intervention took a systems view of impacts: the project aimed to improve water supply to urban areas while maintaining flows to irrigators through enhanced wastewater reuse. The case adds valuable evidence on the impacts of large infrastructure investments and illustrates well the challenges of capturing spillovers, mitigating study contamination, maintaining statistical power, and determining overall welfare effects, in situations involving diverse market and nonmarket impacts. These limitations notwithstanding, the case highlights the high value of conducting IEs, and why applied researchers should not give up on pragmatic and interdisciplinary collaborations to evaluation in the face of complex interventions.

Impact Evaluation of Water Infrastructure Investments: Methods, Challenges and Demonstration From a Large‐Scale Urban Improvement in Jordan

“…Key informant interviews with utility personnel helped to support these explanations; several senior utility operators noted that water “rotations”—periods of service to different neighborhoods within Zarqa—were adjusted following the Compact to allow the benefits of the improvements to be shared more equally across households in improved and unimproved areas. Another area of potential spillovers was in maintenance; because servicing needs declined in WNP areas, effort could be reallocated to non‐Compact areas (Jeuland, Pucilowski, et al., 2020).…”

“…Below, we summarize the main results among beneficiaries in Zarqa (households and small enterprises), farmers in the Jordan Valley, and finally discuss the relative utility performance measures. Additional details can be found in Jeuland, Pucilowski, et al (2020).…”

“…We demonstrated the approach with discussion of a multi‐faceted evaluation implemented to assess the outcomes attributable to the Jordan Compact, a highly integrated project that combined water and wastewater investments, with the goal of reducing water scarcity and enhancing economic opportunity in Zarqa, Jordan. Overall, the evaluation identified a number of positive economic impacts, though results in some domains fell short of expectations (Jeuland, Pucilowski, et al., 2020). On the positive side, there was clear evidence of improved service and reduced water loss where the water network was rehabilitated.…”

“…The evaluation team deemed several additional channels and key assumptions to be potentially consequential (Jeuland, Pucilowski, et al., 2020). First, some hypothesized that households might experience changes in health and general well‐being arising from shifts in sourcing and in consumption levels or other behaviors targeted in the outreach campaigns organized around the JC.…”

“…These growth trends and water losses have been particularly high in urban Zarqa, the region targeted by the infrastructure investment we consider. For example, non‐revenue water (NRW—defined as the sum of unbilled authorized consumption (e.g., water for public uses) plus apparent losses (customer meter inaccuracies, unauthorized consumption and systematic data handling errors) plus real losses (system leakage and storage tank overflows)) has been estimated to exceed 50% over the recent period, well above global and national averages (Jeuland, Pucilowski, et al., 2020).…”