Recent evaluations have begun to use qualitative data in a manner that helps improve the quality and relevance of studies through the inferences that are drawn from them, and their applicability to policy makers and programme implementers. This paper reviews this work and identifies good practices to integrate qualitative methods into quantitative impact evaluations (IEs) and systematic reviews (SRs). Using recent literature on the characteristics of such practices, we developed two tools to assess the methodological rigour and mixed methods integration of 40 IEs and 7 SRs, drawing upon previous approaches. Our findings are that successful mixed methods quantitative impact evaluations: (1) provide a clear rationale for integration of methods; (2) deploy multidisciplinary teams; (3) provide adequate documentation; and (4) acknowledge limitations to the generalisability of qualitative and quantitative findings. Successful integration tended to improve mixed methods impact evaluations by collecting better data to inform the study design and findings, which helped contextualise quantitative findings. Our main observation on the integration of mixed methods in the systematic reviews is that mixed methods systematic reviews bringing together literatures that answer different questions can go beyond the 'sum of their parts' to provide holistic answers about development effectiveness. The findings of this study inform several recommendations to improve the conduct and reporting of mixed methods impact evaluations and systematic reviews.

ObjectivesTo investigate differences in reported open defecation between a question about latrine use or open defecation for every household member and a household-level question.SettingRural India is home to most of the world’s open defecation. India’s Demographic and Health Survey (DHS) 2015–2016 estimates that 54% of households in rural India defecate in the open. This measure is based on a question asking about the behaviour of all household members in one question. Yet, studies in rural India find substantial open defecation among individuals living in households with latrines, suggesting that household-level questions underestimate true open defecation.ParticipantsIn 2018, we randomly assigned latrine-owning households in rural parts of four Indian states to receive one of two survey modules measuring sanitation behaviour. 1215 households were asked about latrine use or open defecation individually for every household member. 1216 households were asked the household-level question used in India’s DHS: what type of facility do members of the household usually use?ResultsWe compare reported open defecation between households asked the individual-level questions and those asked the household-level question. Using two methods for comparing open defecation by question type, the individual-level question found 20–21 (95% CI 16 to 25 for both estimates) percentage points more open defecation than the household-level question, among all households, and 28–29 (95% CI 22 to 35 for both estimates) percentage points more open defecation among households that received assistance to construct their latrines.ConclusionsWe provide the first evidence that individual-level questions find more open defecation than household-level questions. Because reducing open defecation in India is essential to meeting the Sustainable Development Goals, and exposure to open defecation has consequences for child mortality and development, it is essential to accurately monitor its progress.Trial registration numberRegistry for International Development Impact Evaluations (5b55458ca54d1).

The International Initiative for Impact Evaluation (3ie) is an international grant-making NGO promoting evidence-informed development policies and programmes. We are the global leader in producing and synthesising high-quality evidence of what works, for whom, how, why and at what cost. We believe that better and policy-relevant evidence will help make development more effective and improve people's lives. 3ie scoping papers 3ie thematic window grant programmes typically start with a consultative process that includes a scoping study that identifies the current state of impact evaluation evidence in a particular sector. Scoping studies lay out the landscape of what is known and help identify priority policy questions. Although thematic windows are conducted in response to donor demand, this consultation process gives developing country stakeholders a voice in setting key priorities and identifying research questions.

Recent books about evaluations to study the quantitative impact of development programs and projects typically devote a chapter or two of the need to complement the analysis with other methods – specifically qualitative techniques. They often cite how qualitative techniques help explain the reason for positive or negative quantitative results. This is key if the one is to draw conclusions for accountability or for learning to improve future program design. Or they explain how qualitative work is critical to make sure that quantitative data are collected in the right way. Despite these textbook recommendations, there has been a wide range of experiences in how using both quantitative and qualitative methods have affected the overall quality of evaluations. In many cases, the qualitative analysis consists mostly of quotes to justify findings from the quantitative work. While this helps provide context, there is not much value-added beyond making the an otherwise ‘dry’ quantitative presentation more interesting. Some recent evaluations have begun to change this practice and have arguably improved the quality of impact evaluations in terms of their relevance, the inferences that are drawn from them and their applicability to policymakers and programme implementers. This includes the use of innovative techniques to form the specific evaluative questions being asked and tested, to gather the right type of data and information on outcomes and intermediating variables, to explain findings and to disseminate them to the appropriate decision-makers. This paper will review this work. It will canvass a purposeful sample of experts from a variety of disciplines to gather the success stories, and where apparently well-planned approaches have failed to add the value expected of them. It will then draw lessons for future evaluations as a basis for guidance on the use of mixed methods.

ObjectiveThe study aims to check the reusability of impact evaluation data by verifying the results presented in published 3ie reports. In order to verify results, we conduct push button replications on the original data and code submitted by the authors.MethodsWe use the push button replication protocol developed at 3ie to determine the level of comparability of the replication results to the original findings. Our sample includes closed 3ie-funded impact evaluations commissioned between 2008 and 2018.ResultsOf the 74 studies in our sample, we successfully reproduced results from 38 studies (51%). 24 (32%) studies were categorized as incomplete and 12 (16%) studies were categorized as having major differences. The cumulative replication rate in 2018 increased to 51%, as compared to the below-40% replication rate in previous years. Overall, on average, it took about 3 hours to complete the replication of a single impact evaluation. ConclusionEvidence from impact evaluations are credible when it is verifiable. Our findings suggest that greater attention is needed to ensure the reliability and reusability of evidence. We recommend push button replications as a tested method to ascertain the credibility of findings.