Stable Prediction across Unknown Environments

Abstract

In many machine learning applications, the training distribution used to learn a probabilistic classifier differs from the testing distribution on which the classifier will be used to make predictions. Traditional methods correct the distribution shift by reweighting the training data with the ratio of the density between test and training data. But in many applications training takes place without prior knowledge of the testing. Recently, methods have been proposed to address the shift by learning causal structure, but they rely on the diversity of multiple training data to a good performance, and have complexity limitations in high dimensions. In this paper, we propose a novel Deep Global Balancing Regression algorithm to jointly optimize a deep auto-encoder model and a global balancing model for stable prediction across unknown environments. The global balancing model constructs balancing weights that facilitate estimating of partial effects of features, a problem that is challenging in high dimensions, and thus helps to identify stable, causal relationships between features and outcomes. The deep auto-encoder model is designed to reduce the dimensionality of the feature space, thus making global balancing easier. We show, both theoretically and with empirical experiments, that our algorithm can make stable predictions across unknown environments.1. We obtain X·,−j in experiment by setting the value of j th variable in X as zero.2. We use sigmoid function σ(x) = 1 1+exp(−x) as non-linear activation function.

EKT: Exercise-Aware Knowledge Tracing for Student Performance Prediction

“…A qualified model for student performance prediction should have good results from both regression and classification perspectives. In this paper, we evaluated the prediction performance of all models using four widely-used metrics in the domain [13], [24], [49], [50], [56]. From the regression perspective, we selected Mean Absolute Error (MAE) and Root Mean Square Error (RMSE), to quantify the distance between predicted scores and the actual ones.…”

Stable Prediction with Model Misspecification and Agnostic Distribution Shift

“…The joint distribution of features and outcomes on (X, Y ) can change across environments: P e XY = P e XY for e, e ∈ E. In this paper, our goal is to learn a predictive model for stable prediction with model misspecification and agnostic distribution shift. To measure its performance on stable prediction problem, we adopt the Average Error and Stability Error in (Kuang et al 2018) as:…”

“…where |E| refers to the number of test environments, and RM SE(D e ) represents the Root Mean Square Error of a predictive model on dataset D e . Actually, Average Error and Stability Error refer to the mean and variance of predictive error over all possible environments e ∈ E. Then, the stable prediction problem (Kuang et al 2018) is defined as: Problem 1 (Stable Prediction) Given one training environment e ∈ E with dataset D e = (X e , Y e ), the task is to learn a predictive model to predict across unknown environment E with not only small Average Error but also small Stability Error.…”

“…Moreover, they do not consider the interaction of distribution shift and model misspecification. Recently, some papers (Kuang et al 2018;Shen et al 2018) were proposed to address stable prediction problem using methods drawn from the literature on causal inference, achieving improved performance. But they did not consider the model misspecification and their algorithms were restricted to the predictive setting with binary predictors and binary response variable.…”

DeVLBert

“…A fundamental requirement for out-of-domain transfer learning is to mitigate the biases from the pretraining data [49], which may be useful for the in-domain testing but harmful for out-of-domain testing [19] due to the spurious correlation [34]. To verify such existence of the correlation biases, we follow [49] to conduct a toy experiment on Conceptual Caption dataset.…”