Quantitative errors in the Cochrane review on "Physical interventions to interrupt or reduce the spread of respiratory viruses"

Abstract

The COVID-19 pandemic has brought a heightened sense of urgency in the scientific community regarding the need to advance understanding and prevention of pathogen transmission, particularly concerning infectious airborne particles and the utility of various preventive strategies in reducing the risk of infection. There are extensive studies validating scientific understanding about the behavior of larger (droplets) and smaller (aerosols) particles in disease transmission and the dosimetry of particles in the respiratory track. Similarly, modalities for respiratory protection against particles in the size range spanned by infectious particles, such as N95 respirators, are available and known to be efficacious with tested standards for harm reduction across environments including physical, chemical and biological hazards. Even though multiple studies also confirm their protective effect when adopted in healthcare and public settings for infection prevention, overall, studies of protocols of their adoption over the last several decades in both clinical trials and observational studies have not provided as clear an understanding. Here we demonstrate that these studies are strongly biased towards the null by infections resulting from transmission outside of the investigated environments and study participants. Such study limitations are frequently mis-stated as not influencing the conclusions of research on respiratory protection. The reason for the failure to properly analyze the studies is that the standard analytical equations used do not correctly represent the random variables that play a role in the study results. By correcting the mathematical representation and the equations that result from them, we demonstrate that conclusions drawn from these studies are strongly biased and much more uncertain than is acknowledged, providing almost no useful information. Even with all these limitations, we show that existing results, when outcome measures are properly analyzed, consistently point to the benefit of precautionary measures such as N95 respirators over medical masks, and masking over its absence. We also show that correcting manifest errors of widely reported meta-analyses also leads to statistically significant estimates. Our results have implications for the design of studies and analyses on the effectiveness of respiratory protection and on using existing evidence for policy guidelines for infection control.

Medical masks versus N95 respirators for preventing COVID-19 among health care workers: A secondary analysis of findings inconsistent with prior understanding reflects the expected inferiority of medical masks.

“…While the authors have already incorporated the trial into a controversial (83,84,163) Cochrane review arguing there are "…no differences in clinical effectiveness…" between medical masks and N95,(82) analysis of their own data using the prespecified per protocol approach and noninferiority criteria does not support their claim of noninferiority for MM in comparison to N95. Late-stage enlargement of Δ not recorded in the trial registry until after the trial was complete, and omission of the originally specified analysis resulted in a complete reversal of the trial outcome, aligning with the authors' competing interests.…”

“…The effects of the trial's many shortcomings were not evenly distributed but consistently (see Appendix) biased the trial towards a finding that MM were noninferior to N95, aligned with the authors' serious unreported conflicts of interest. While widely publicized by the authors (124,125) and already being encoded into nominally authoritative infection control literature (82,83) and practice, (81,85) the primary finding was obtained only via multiple significant alterations to the conduct, analysis and criteria of the trial, none of which were prespecified in the trial registry. In at least the majority of cases the impacts of those alterations were predictable by the Trial Steering Committee at the time they were made.…”

“…The initial proposal to employ discontinuous use of N95 against a suspected aerosol hazard is a fatal flaw by itself, and is also present in two influenza studies often cited as evidence against efficacy of N95: one by Loeb in 2009 (162) and one by Radonovich in 2019. (200) Their uncritical embrace along with the flawed and controversial Cochrane review (83,84,163) in efforts to downgrade the level of protection required for SARS-CoV-2 highlights a need to halt these efforts until good faith can be confirmed (as opposed to being e.g. an attempt to construct a retroactive justification for prior failures to provide adequate PPE).…”

“…Both of these alterations have the effect of reducing any differences between MM and N95 groups, and as they were not accounted for, they introduce significant errors into the interpretation. A recent examination by Bar-Yam and colleagues of methodological errors (83) in the Cochrane review (82,163) into which the Loeb study was rapidly incorporated includes a thorough quantitative assessment of the magnitude of the biases towards noninferiority introduced into the Loeb trial specifically by factors such as inconsistent and/or discontinuous use of PPE and contributions of community-acquired infection. This analysis demonstrates that despite the reported finding of noninferiority using the altered Δ criterion of a two-fold increase in relative to N95 as the acceptable bound, the actual bound on increased risk from MM in the Loeb study is not less than a tenfold increase relative to N95.…”

“…This analysis demonstrates that despite the reported finding of noninferiority using the altered Δ criterion of a two-fold increase in relative to N95 as the acceptable bound, the actual bound on increased risk from MM in the Loeb study is not less than a tenfold increase relative to N95. (83) Ambiguity around the dates of cessation of enrolment vs follow-up raises the question whether any participants were abandoned to follow-up due to the termination of the study in Canada, and how that might have affected the results. Without access to the anonymized data, any impact of the authors' decisions to reduce follow-up from 12 weeks to 10 (included in the Dec. 21, 2022 post-publication modification to the registered study, (179) but back-dated to May 4, 2020) is also unclear.…”