Problems with Evidence Assessment in COVID-19 Health Policy Impact Evaluation: A systematic review of study design and evidence strength

Abstract

IntroductionThe impact of policies on COVID-19 outcomes is one of the most important questions of our time. Unfortunately, there are substantial concerns about the strength and quality of the literature examining policy impacts. This study systematically assessed the currently published COVID-19 policy impact literature for a checklist of study design elements and methodological issues.MethodsWe included studies that were primarily designed to estimate the quantitative impact of one or more implemented COVID-19 policies on direct SARS-CoV-2 and COVID-19 outcomes. After searching PubMed for peer-reviewed articles published on November 26 or earlier and screening, all studies were reviewed by three reviewers independently and in consensus. The review tool was based on review guidance for assessing COVID-19 health policy impact evaluation analyses, including first identifying the assumptions behind the methods used, followed by assessing graphical display of outcomes data, functional form for the outcomes, timing between policy and impact, concurrent changes to the outcomes, and an overall rating.ResultsAfter 102 articles were identified as potentially meeting inclusion criteria, we identified 36 published articles that evaluated the quantitative impact of COVID-19 policies on direct COVID-19 outcomes. The majority (n=23/36) of studies in our sample examined the impact of stay-at-home requirements. Nine studies were set aside due to inappropriate study design (n=8 pre/post; n=1 cross-section), and 27 articles were given a full consensus assessment. 20/27 met criteria for graphical display of data, 5/27 for functional form, 19/27 for timing between policy implementation and impact, and only 3/27 for concurrent changes to the outcomes. Only 1/27 studies passed all of the above checks, and 4/27 were rated as overall appropriate. Including the 9 studies set aside, we found that only four (or by a stricter standard, only one) of the 36 identified published and peer-reviewed health policy impact evaluation studies passed a set of key design checks for identifying the causal impact of policies on COVID-19 outcomes.DiscussionThe current literature directly evaluating the impact of COVID-19 policies largely fails to meet key design criteria for useful inference. This may be partially due to the circumstances for evaluation being particularly difficult, as well as a context with desire for rapid publication, the importance of the topic, and weak peer review processes. Importantly, weak evidence is non-informative and does not indicate how effective these policies were on COVID-19 outcomes.

US Trends in COVID-19–Associated Hospitalization and Mortality Rates Before and After Reopening Economies

“…We conducted several sensitivity analyses to confirm the robustness of our estimates. 20 First, we tested alternative washout periods of 8 days (from reopening), corresponding to the 25th percentile of incubation, and 15 days (from reopening), corresponding to the 75th percentile of the incubation period (from infection to hospitalization). 21 , 22 Second, to examine whether we could estimate state reopening associations similar in magnitude to ours by chance, we randomized the timing of state reopenings to alternative pseudo start dates in the preintervention time continuum.…”

The impact of school closures on adolescent health-related outcomes during the COVID-19 pandemic: A natural experiment in South Korea

“…nation-wide restaurant closures) limiting causal inference(18). Furthermore, cooccurrence of simultaneous policy changes at the same time and place will unfortunately eliminate the chance to separate the effect of interest(19). In South Korea, school closure was implemented in a more principled manner, where the school opening policy was consistent throughout the year(20).…”

“…Changes in physical activity was the only change that was found significant. As these studies rely on simple pre/post comparisons or retrospective cohort designs, they are prone to unobserved confounding(9, 17, 19). This underscores the importance of properly designed studies that are robust to such biases.…”

Evaluating the impact of keeping indoor dining closed on COVID-19 rates among large US cities: a quasi-experimental design

“…Understanding which Non-Pharmaceutical Interventions (NPIs) reduce transmission is crucial for balancing public health and economic and social cost from COVID-19. Previous research suggests that mask mandates 2 , sick leave policies 3 , and shelter in place orders reduce the spread of COVID-19 4 , although the quality of studies concerning the causal evaluation of NPIs is mixed 5 . Many analyses evaluate state-level policies, masking substantial city-and county-level heterogeneity.…”