ZIN: When and How to Learn Invariance by Environment Inference?

Abstract

It is commonplace to encounter heterogeneous data, of which some aspects of the data distribution may vary but the underlying causal mechanisms remain constant. When data are divided into distinct environments according to the heterogeneity, recent invariant learning methods have proposed to learn robust and invariant models based on this environment partition. It is hence tempting to utilize the inherent heterogeneity even when environment partition is not provided. Unfortunately, in this work, we show that learning invariant features under this circumstance is fundamentally impossible without further inductive biases or additional information. Then, we propose a framework to jointly learn environment partition and invariant representation, assisted by additional auxiliary information. We derive sufficient and necessary conditions for our framework to provably identify invariant features under a fairly general setting. Experimental results on both synthetic and real world datasets validate our analysis and demonstrate an improved performance of the proposed framework over existing methods. Finally, our results also raise the need of making the role of inductive biases more explicit in future works, when considering learning invariant models without environment partition.

Learning Optimal Features via Partial Invariance

“…For our choice of learning rate, number of iterations and optimizer and annealing iterations, we refer to ((Lin, Zhu, and Cui 2022)). While the reported results were for λ = 10 2 , we verified similar trends for λ = 10 (Lin, Zhu, and Cui 2022) .…”

“…As pre-processing step, we drop all non-numerical features, dropping samples with missing values and normalizing each feature and the price label to zero mean, unit variance and the samples, {X i , y i } i ∈ (R 32 × R). Experiment Setup To adapt this task to OoD prediction, following (Lin, Zhu, and Cui 2022), we manually split the training data-set into 10-year segments and use the house year built as a meta-data for partitioning, with the intuition being that factors affecting house prices change over time with societal perceptions. For prediction, we consider a linear regression model for the task.…”

Frustratingly Easy Environment Discovery for Invariant Learning

“…Recent studies have addressed biases by learning invariances in training data. Motivated by casual discovery, IRM [3] and its variants [25,[30][31][32][33] learn a representation such that the optimal classifier built on top is the same for all training environments. LISA [34] also learns invariant predictors via selective mix-up augmentation across different environments.…”

When Does Group Invariant Learning Survive Spurious Correlations?

“…In recent works, invariant learning is extended to the scenario without a priori environment labels, but with knowledge on spurious correlations in the training data [34; 8]. Such knowledge is proven to be necessary in this setting [18]. They are utilized to split the training data into groups, which are supposed to encode variations of spurious information so that they can be avoided by learning the invariance.…”

“…Therefore, we need specific theoretical analysis under the setting of group-IL. In a recent work, Lin et al [18] derived the sufficient and necessary assumptions for their proposed algorithm. However, in this paper we focus on group criteria for general group-IL methods.…”