Causally Regularized Learning with Agnostic Data Selection Bias

Abstract

Most of previous machine learning algorithms are proposed based on the i.i.d. hypothesis. However, this ideal assumption is often violated in real applications, where selection bias may arise between training and testing process. Moreover, in many scenarios, the testing data is not even available during the training process, which makes the traditional methods like transfer learning infeasible due to their need on prior of test distribution. Therefore, how to address the agnostic selection bias for robust model learning is of paramount importance for both academic research and real applications. In this paper, under the assumption that causal relationships among variables are robust across domains, we incorporate causal technique into predictive modeling and propose a novel Causally Regularized Logistic Regression (CRLR) algorithm by jointly optimize global confounder balancing and weighted logistic regression. Global confounder balancing helps to identify causal features, whose causal effect on outcome are stable across domains, then performing logistic regression on those causal features constructs a robust predictive model against the agnostic bias. To validate the effectiveness of our CRLR algorithm, we conduct comprehensive experiments on both synthetic and real world datasets. Experimental results clearly demonstrate that our CRLR algorithm outperforms the state-of-the-art methods, and the interpretability of our method can be fully depicted by the feature visualization.

Generalizing Graph Neural Networks on Out-Of-Distribution Graphs

“…These methods mainly bring the ideas from the causal effect estimation (Angrist & Imbens, 1995) into machine learning models. Particularly, Shen et al (Shen et al, 2018) propose a global confounding balancing regularizer that helps the logistic regression model to identify causal features, whose causal effect on outcome are stable across domains. To make the confounder balancing much easier in high-dimensional scenario, Kuang et al utilize the autoencoder to encode the high-dimensional features into low-dimensional representation.…”

A Theoretical Analysis on Independence-driven Importance Weighting for Covariate-shift Generalization

“…Stable learning Stable learning algorithms can be considered as a feature selection mechanism according to the regression coefficients . Motivated by the literature of variable balancing methods [Hainmueller, 2012;Zubizarreta, 2015;Athey et al, 2016], Shen et al [2018] proposed to consider all the variables as the treatment and learn a set of weights for all of available samples to remove the confounding bias from data distribution. Specifically, a global balancing loss is proposed as a regularizer which can be easily plugged into machine learning models.…”

“…Shen et al [2020a] further proposed to recover the latent cluster structures among variables using unlabeled data and proved decorrelating the variables between clusters instead of each other sufficient to achieve a stable estimation while preventing the variance inflation. Recently, proposed a framework named StableNet, which extends former linear stable learning frameworks [Shen et al, 2018;Shen et al, 2020b] to incorporate deep models. StableNet adopted Random Fourier Features (RFF) [Rahimi et al, 2007] to eliminate non-linear dependences between features sufficiently.…”

Meta-Learning Causal Feature Selection for Stable Prediction

“…The idea is that causal models are more robust to changes in real-world datasets. Causality is integrated into the classifier to estimate the effect of every dimensional feature for the label by treating features as intervention variables and labels as outcome variables [13]. However, these methods require to learn a set of sample weights for every batch of samples during training, which makes it time-consuming and takes up a lot of computing resources.…”