Generalizing Graph Neural Networks on Out-Of-Distribution Graphs

Abstract

Graph Neural Networks (GNNs) are proposed without considering the agnostic distribution shifts between training graphs and testing graphs, inducing the degeneration of the generalization ability of GNNs on Out-Of-Distribution (OOD) settings. The fundamental reason for such degeneration is that most GNNs are developed based on the I.I.D hypothesis. In such a setting, GNNs tend to exploit subtle statistical correlations existing in the training set for predictions, even though it is a spurious correlation. This learning mechanism inherits from the common characteristics of machine learning approaches. However, such spurious correlations may change in the wild testing environments, leading to the failure of GNNs. Therefore, eliminating the impact of spurious correlations is crucial for stable GNN models. To this end, in this paper, we argue that the spurious correlation exists among subgraph-level units and analyze the degeneration of GNN in causal view. Based on the causal view analysis, we propose a general causal representation framework for stable GNN, called StableGNN. The main idea of this framework is to extract high-level representations from raw graph data first and resort to the distinguishing ability of causal inference to help the model get rid of spurious correlations. Particularly, to extract meaningful high-level representations, we exploit a differentiable graph pooling layer to extract subgraph-based representations by an end-to-end manner. Furthermore, inspired by the confounder balancing techniques from causal inference, based on the learned high-level representations, we propose a causal variable distinguishing regularizer to correct the biased training distribution by learning a set of sample weights. Hence, GNNs would concentrate more on the true connection between discriminative substructures and labels. Extensive experiments are conducted on both synthetic datasets with various distribution shift degrees and eight real-world OOD graph datasets. The results well verify that the proposed model StableGNN not only outperforms the state-of-the-arts but also provides a flexible framework to enhance existing GNNs. In addition, the interpretability experiments validate that StableGNN could leverage causal structures for predictions.

Learning to Reweight for Generalizable Graph Neural Network

“…However, in practice, it might be challenging to satisfy this ideal hypothesis. Recent research (Fan et al 2021;) studies how well GNNs generalize outside the training distribution. Several studies concentrate on size generalization ability to make GNNs function effectively on testing graphs whose size distribution is different from that of training graphs.…”

“…We suggest that not all correlations should be eliminated, in contrast to prior techniques (Fan et al 2021;, which aggressively decorrelate all connections across graph representations. Such an aggressive objective may result in an issue with an overly-reduced sample size (Martino, Elvira Llorente et al 2022), which hampers the generalization ability of GNNs.…”

“…However, the majority of these approaches are suggested in linear settings. GNNs combine heterogeneous data from node features and graph topological structures, resulting in the existence of intricate and unrecognized non-linear relationships across representations (Fan et al 2021;. Non-linear dependencies on graph data cannot be removed using linear sample reweighting approaches.…”

“…Non-linear dependencies on graph data cannot be removed using linear sample reweighting approaches. Recent studies propose non-linear decorrelation methods for graph tasks (Fan et al 2021;. They attempt to eliminate the dependencies between all the variables of the graph representation through a set of learned sample weights.…”

Partitioning Distributed Compute Jobs with Reinforcement Learning and Graph Neural Networks

“…This study ran PAC-ML in an environment which had the same load rate, β distribution, cluster network size, and job computation graphs at train and test time. An interesting research question would be whether PAC-ML would be able to learn on one set (or a distribution) of these parameters and then generalise to a new set at test time, or if it would need to leverage existing or new state-of-the-art methods in GNN (Knyazev et al, 2019;Garg et al, 2020;Fan et al, 2021) and RL (Cobbe et al, 2019;Wang et al, 2020;Kirk et al, 2021) generalisation.…”