Uncovering Sociological Effect Heterogeneity Using Tree-Based Machine Learning

Abstract

Individuals do not respond uniformly to treatments, such as events or interventions. Sociologists routinely partition samples into subgroups to explore how the effects of treatments vary by selected covariates, such as race and gender, on the basis of theoretical priors. Data-driven discoveries are also routine, yet the analyses by which sociologists typically go about them are often problematic and seldom move us beyond our biases to explore new meaningful subgroups. Emerging machine learning methods based on decision trees allow researchers to explore sources of variation that they may not have previously considered or envisaged. In this article, the authors use tree-based machine learning, that is, causal trees, to recursively partition the sample to uncover sources of effect heterogeneity. Assessing a central topic in social inequality, college effects on wages, the authors compare what is learned from covariate and propensity score–based partitioning approaches with recursive partitioning based on causal trees. Decision trees, although superseded by forests for estimation, can be used to uncover subpopulations responsive to treatments. Using observational data, the authors expand on the existing causal tree literature by applying leaf-specific effect estimation strategies to adjust for observed confounding, including inverse propensity weighting, nearest neighbor matching, and doubly robust causal forests. We also assess localized balance metrics and sensitivity analyses to address the possibility of differential imbalance and unobserved confounding. The authors encourage researchers to follow similar data exploration practices in their work on variation in sociological effects and offer a straightforward framework by which to do so.

Leveraging Machine Learning to Estimate Effect Modification

“…They optimize the accuracy of individual-level predictions and directly output predictions at the individual level as opposed to the aggregate level of substantive interest. The causal forests, a variant of Generalized Random Forests (GRF), (Athey, Tibshirani, and Wager 2019; Wager 2019) is a prominent example in this category and has been applied in multiple social science settings (Brand et al 2021;Daoud and Johansson 2019;Knittel and Stolper 2019;Tiffin 2019). Other examples include the R-learner (Nie and Wager 2020), the X-learner (Künzel et al 2019), and the Modified Covariate Method (Chen et al 2017;Tian et al 2014).…”

“…Apart from effect modification and individual-level effect prediction, there is yet another goal that may interest empirical researchers. Researchers have also employed ML methods to discover important effect modifiers that are previously ignored and may lead to novel insights about effect heterogeneity (Brand et al 2021). For example, causal forests output a variable importance metric that can be used for this purpose.…”

“…Using National Longitudinal Survey of Youth 1979 (NLSY79), we investigate modification of the effect of a college degree on adult income along the dimensions of gender and income origin. Studying college effect heterogeneity provides insights about higher education and inform the influence of potential policies that could affect the college admission process (Brand et al 2021;Brand and Xie 2010;Forster, van de Werfhorst, and Leopold 2021;Heckman, Humphries, and Veramendi 2018). In addition, the pattern of effect modification also implies whether college education serves as an equalizer that reduces inequality in terms of the focal modifier.…”

“…For example, methodological adaptations of a wide variety of ML methods have appeared in life course research (Billari, Fürnkranz, and Prskawetz 2006), criminology (Baćak and Kennedy 2019), survey design (Fu, Guo, and Land 2018), text and image analysis (Rona-Tas et al 2019;Zhang and Pan 2019), causal inference (Brand et al 2021;Liu 2021;Torrats-Espinosa 2021), log-linear modelling (Bucca and Urbina 2019), and general evaluation of the predictability of life and physical outcomes (Daoud, Kim, and Subramanian 2019;Salganik et al 2020).…”

“…In agreement with Zhou (2019), Fiel (2020), and Forster et al (2021, our finding also casts doubt on the great equalizer role of college, which would require higher treatment effect among disadvantaged groups (cf. Brand et al 2021;Torche 2011). If anything, high-income-origin is associated with a slightly higher return to college education.…”