Recently, there has been growing interest in using machine learning methods for causal inference due to their automatic and flexible ability to model the propensity score and the outcome model. However, almost all the machine learning methods for causal inference have been studied under the assumption of no unmeasured confounding and there is little work on handling omitted/unmeasured variable bias. This paper focuses on a machine learning method based on random forests known as Causal Forests and presents five simple modifications for tuning Causal Forests so that they are robust to cluster-level unmeasured confounding. Our simulation study finds that adjusting the default tuning procedure with the propensity score from fixed effects logistic regression or using variables that are centered to their cluster means produces estimates that are more robust to cluster-level unmeasured confounding. Also, when these parametric propensity score models are mis-specified, our modified machine learning methods remain robust to bias from cluster-level unmeasured confounders compared to existing parametric approaches based on propensity score weighting. We conclude by demonstrating our proposals in a real data study concerning the effect of taking an eighth-grade algebra course on math achievement scores from the Early Childhood Longitudinal Study.

There is a growing interest in using machine learning (ML) methods for causal inference due to their (nearly) automatic and flexible ability to model key quantities such as the propensity score or the outcome model. Unfortunately, most ML methods for causal inference have been studied under single-level settings where all individuals are independent of each other and there is little work in using these methods with clustered or nested data, a common setting in education studies. This paper investigates using one particular ML method based on random forests known as Causal Forests to estimate treatment effects in multilevel observational data. We conduct simulation studies under different types of multilevel data, including two-level, three-level, and cross-classified data. Our simulation study shows that when the ML method is supplemented with estimated propensity scores from multilevel models that account for clustered/hierarchical structure, the modified ML method outperforms pre-existing methods in a wide variety of settings. We conclude by estimating the effect of private math lessons in the Trends in International Mathematics and Science Study data, a large-scale educational assessment where students are nested within schools.

Introduction Available evidence is mixed concerning associations between smoking status and COVID-19 clinical outcomes. Effects of nicotine replacement therapy (NRT) and vaccination status on COVID-19 outcomes in smokers are unknown. Methods Electronic health record data from 104 590 COVID-19 patients hospitalized February 1, 2020 to September 30, 2021 in 21 U.S. health systems were analyzed to assess associations of smoking status, in-hospital NRT prescription, and vaccination status with in-hospital death and ICU admission. Results Current (n = 7764) and never smokers (n = 57 454) did not differ on outcomes after adjustment for age, sex, race, ethnicity, insurance, body mass index, and comorbidities. Former (vs never) smokers (n = 33 101) had higher adjusted odds of death (aOR, 1.11; 95% CI, 1.06–1.17) and ICU admission (aOR, 1.07; 95% CI, 1.04–1.11). Among current smokers, NRT prescription was associated with reduced mortality (aOR, 0.64; 95% CI, 0.50–0.82). Vaccination effects were significantly moderated by smoking status; vaccination was more strongly associated with reduced mortality among current (aOR, 0.29; 95% CI, 0.16–0.66) and former smokers (aOR, 0.47; 95% CI, 0.39–0.57) than for never smokers (aOR, 0.67; 95% CI, 0.57, 0.79). Vaccination was associated with reduced ICU admission more strongly among former (aOR, 0.74; 95% CI, 0.66–0.83) than never smokers (aOR, 0.87; 95% CI, 0.79–0.97). Conclusions Former but not current smokers hospitalized with COVID-19 are at higher risk for severe outcomes. SARS-CoV-2 vaccination is associated with better hospital outcomes in COVID-19 patients, especially current and former smokers. NRT during COVID-19 hospitalization may reduce mortality for current smokers. Implications Prior findings regarding associations between smoking and severe COVID-19 disease outcomes have been inconsistent. This large cohort study suggests potential beneficial effects of nicotine replacement therapy on COVID-19 outcomes in current smokers and outsized benefits of SARS-CoV-2 vaccination in current and former smokers. Such findings may influence clinical practice and prevention efforts and motivate additional research that explores mechanisms for these effects.

Main objective There is limited information on how patient outcomes have changed during the COVID-19 pandemic. This study characterizes changes in mortality, intubation, and ICU admission rates during the first 20 months of the pandemic. Study design and methods University of Wisconsin researchers collected and harmonized electronic health record data from 1.1 million COVID-19 patients across 21 United States health systems from February 2020 through September 2021. The analysis comprised data from 104,590 adult hospitalized COVID-19 patients. Inclusion criteria for the analysis were: (1) age 18 years or older; (2) COVID-19 ICD-10 diagnosis during hospitalization and/or a positive COVID-19 PCR test in a 14-day window (+/- 7 days of hospital admission); and (3) health system contact prior to COVID-19 hospitalization. Outcomes assessed were: (1) mortality (primary), (2) endotracheal intubation, and (3) ICU admission. Results and significance The 104,590 hospitalized participants had a mean age of 61.7 years and were 50.4% female, 24% Black, and 56.8% White. Overall risk-standardized mortality (adjusted for age, sex, race, ethnicity, body mass index, insurance status and medical comorbidities) declined from 16% of hospitalized COVID-19 patients (95% CI: 16% to 17%) early in the pandemic (February-April 2020) to 9% (CI: 9% to 10%) later (July-September 2021). Among subpopulations, males (vs. females), those on Medicare (vs. those on commercial insurance), the severely obese (vs. normal weight), and those aged 60 and older (vs. younger individuals) had especially high mortality rates both early and late in the pandemic. ICU admission and intubation rates also declined across these 20 months. Conclusions Mortality, intubation, and ICU admission rates improved markedly over the first 20 months of the pandemic among adult hospitalized COVID-19 patients although gains varied by subpopulation. These data provide important information on the course of COVID-19 and identify hospitalized patient groups at heightened risk for negative outcomes. Trial registration ClinicalTrials.gov Identifier: NCT04506528 (https://clinicaltrials.gov/ct2/show/NCT04506528).

Recently, machine learning (ML) methods have been used in causal inference to estimate treatment effects in order to reduce concerns for model mis-specification. However, many ML methods require that all confounders are measured to consistently estimate treatment effects. In this paper, we propose a family of ML methods that estimate treatment effects in the presence of cluster-level unmeasured confounders, a type of unmeasured confounders that are shared within each cluster and are common in multilevel observational studies. We show through simulation studies that our proposed methods are robust from biases from unmeasured cluster-level confounders in a variety of multilevel observational studies. We also examine the effect of taking an algebra course on math achievement scores from the Early Childhood Longitudinal Study, a multilevel observational educational study, using our methods. The proposed methods are available in the CURobustML R package.