Access to safe drinking water: experimental evidence from new water sources in Bangladesh

Abstract

About 3ie The International Initiative for Impact Evaluation (3ie) promotes evidence-informed, equitable, inclusive and sustainable development. We support the generation and effective use of high-quality evidence to inform decision-making and improve the lives of people living in poverty in low-and middle-income countries. We provide guidance and support to produce, synthesise and quality assure evidence of what works, for whom, how, why and at what cost. 3ie impact evaluations 3ie-supported impact evaluations assess the difference a development intervention has made to social and economic outcomes. 3ie is committed to funding rigorous evaluations that include a theory-based design and that use the most appropriate mix of methods to capture outcomes and are useful in complex development contexts. About this report 3ie accepted the final version of the report, Rebuilding the social compact: urban service delivery and property taxes in Pakistan, as partial fulfilment of requirements under grant DPW1.1005 awarded through Development Priorities Window 1. The report is technically sound and 3ie is making it available to the public in this final report version as it was received. No further work has been done. The 3ie technical quality assurance team for this report comprises Francis Rathinam, Neeta Goel, Kanika Jha Kingra and Deeksha Ahuja, an anonymous external impact evaluation design expert reviewer and an anonymous external sector expert reviewer, with overall technical supervision by Marie Gaarder. The 3ie editorial production team for this report comprises Anushruti Ganguly and Akarsh Gupta.

Making and justifying evidence claims: evidence synthesis of impact evaluations and systematic reviews in international development

“…Other justifications included acknowledgement of the fact that these methods were commonly used in the field, or based on previous work (Asfaw et al, 2017;Bett et al, 2018;Gibbs et al, 2020;Tsai et al, 2018). Three other studies argued that the chosen methods were designed to encourage participation (Cocciolo et al, 2020), be adaptable (Li and Liu, 2020), and fill the research gaps (Morten et al, 2020). Four studies claimed that the chosen research methods allowed them to address spillover effects.…”

“…Of the 15 studies which used the robustness of results as a justification for empirical claims, 13 achieved this through the use of explicit statistical modelling techniques, such as sensitivity analysis, testing additional control variables and robust standard errors. Two studies, Béné et al (2020) and Cocciolo et al (2020), presented their results as robust, based on the general rigour and quality of their methods and data collection. Similarly, eight studies checked potential biases or performed balance checks to prove the effective randomisation and internal validity of the evaluation (Aker et al, 2017;Angeles et al, 2019;Armand et al, 2019;Bandiera et al, 2017;Barnett et al, 2018;Diagne and Cabral, 2017;Källander et al, 2021;Pellegrini, 2018).…”

“…Furthermore, theory was used to justify the certainty and impact of claims across several studies (n=7). This was done through a return to the theory of change (Béné et al, 2020;Cocciolo et al, 2020;Parker et al, 2019) or an examination of the consistency of results with established theories (Amirapu et al, 2020;Belissa et al, 2019;Berhane et al, 2017;Gibbs et al, 2020).…”

“…The limitations that were cited were also largely methodological, with the majority (n=20) relating to biases or baseline imbalances. Many included references to the impact of social desirability bias: for example, Cocciolo andothers (2020) acknowledged 'previous research (e.g. Ahuja et al 2010) and our own experience suggest that social desirability bias influences household reports of behaviour with respect to obtaining drinking water ' (p. 36).…”

“…Of 13 studies making claims about the choice of the chosen study design, seven employed quantitative research methods, whereas six studies employed a mixedmethods design. The choice of the study design was often supported by the expected quality or added value of employing the chosen research design to address the research questions on causality (Avdeenko and Frölich, 2019;Bandiera et al, 2017;Brudevold-Newman et al, 2017, Cocciolo et al, 2020. For example, one mixed-methods study evaluating community building and humanitarian aid interventions in Pakistan explained the benefits of the RCT for establishing causality based on the post-intervention differences.…”