Program Evaluation of Population- and System-Level Policies: Evidence for Decision Making

Abstract

Background Policy evaluations often focus on ex post estimation of causal effects on short-term surrogate outcomes. The value of such information is limited for decision making, as the failure to reflect policy-relevant outcomes and disregard for opportunity costs prohibits the assessment of value for money. Further, these evaluations do not always consider all relevant evidence, other courses of action, or decision uncertainty. Methods In this article, we explore how policy evaluation could better meet the needs of decision making. We begin by defining the evidence required to inform decision making. We then conduct a literature review of challenges in evaluating policies. Finally, we highlight potential methods available to help address these challenges. Results The evidence required to inform decision making includes the impacts on the policy-relevant outcomes, the costs and associated opportunity costs, and the consequences of uncertainty. Challenges in evaluating health policies are described using 8 categories: 1) valuation space; 2) comparators; 3) time of evaluation; 4) mechanisms of action; 5) effects; 6) resources, constraints, and opportunity costs; 7) fidelity, adaptation, and level of implementation; and 8) generalizability and external validity. Methods from a broad set of disciplines are available to improve policy evaluation, relating to causal inference, decision-analytic modeling, theory of change, realist evaluation, and structured expert elicitation. Limitations The targeted review may not identify all possible challenges, and the methods covered are not exhaustive. Conclusions Evaluations should provide appropriate evidence to inform decision making. There are challenges in evaluating policies, but methods from multiple disciplines are available to address these challenges. Implications Evaluators need to carefully consider the decision being informed, the necessary evidence to inform it, and the appropriate methods. [Box: see text]

Urban Economic Resilience and Supply Chain Dynamics: Evaluating Monetary Recovery Policies in Global Cities during the Early COVID-19 Pandemic

“…Traditional methods such as social experiments and other econometric tools [51], while valuable, often fail to fully capture the multi-dimensional impacts of crises on urban economies and supply chains. In line with recent methodologies underscoring the need for innovative policy evaluation approaches [52][53][54], our DEA-PCA-tobit model offers a comprehensive and accurate analysis of policy effectiveness in enhancing supply chain resilience and efficiency. Importantly, the introduction of an industry coverage indicator in policy assessment recognises the extensive damage inflicted by the COVID-19 pandemic on various industries and supply chains, underscoring its significance in evaluating policy effectiveness.…”

Supporting the revision of the health benefits package in Uganda: A constrained optimisation approach

“…Among the empirical studies, those which took a more comprehensive view of the health sector rather than a specific disease or programme only applied an “overall” financial constraint facing a health system reflecting the financial cost of all resource inputs (Lofgren et al., 2021; Varghese et al., 2020). While such an approach may be suitable over the long run over which all resources are potentially flexible, in the short run, there are multiple constraints on care both financial and non‐financial (vanBaal et al., 2018; Walker et al., 1177; Revill et al., 2018). There is a need, therefore, for empirical research to demonstrate how other resource constraints can be captured in conjunction with the public health budget constraint to arrive at an optimal HBP.…”

Consensus on the definition and assessment of external validity of randomized controlled trials: A Delphi study

“…External validity is considered an important factor for decision making in health research. 1,2 Although research on external validity has increased in the last decades, 1 there are still many shortcomings and methodological issues in this regard. 3,4 Research has focused on examining various aspects of the internal validity (rather than external validity) of randomized controlled trials (RCTs) (e.g.…”

“…External validity is considered an important factor for decision making in health research 1,2 . Although research on external validity has increased in the last decades, 1 there are still many shortcomings and methodological issues in this regard 3,4 …”