GOOD: A Graph Out-of-Distribution Benchmark

Abstract

Out-of-distribution (OOD) learning deals with scenarios in which training and test data follow different distributions. Although general OOD problems have been intensively studied in machine learning, graph OOD is only an emerging area of research. Currently, there lacks a systematic benchmark tailored to graph OOD method evaluation. In this work, we aim at developing an OOD benchmark, known as GOOD, for graphs specifically. We explicitly make distinctions between covariate and concept shifts and design data splits that accurately reflect different shifts. We consider both graph and node prediction tasks as there are key differences when designing shifts. Overall, GOOD contains 8 datasets with 14 domain selections. When combined with covariate, concept, and no shifts, we obtain 42 different splits. We provide performance results on 7 commonly used baseline methods with 10 random runs. This results in 294 dataset-model combinations in total. Our results show significant performance gaps between in-distribution and OOD settings. Our results also shed light on different performance trends between covariate and concept shifts by different methods. Our GOOD benchmark is a growing project and expects to expand in both quantity and variety of resources as the area develops. The GOOD benchmark can be accessed via https://github.com/divelab/GOOD/. * Equal contributions Preprint. Under review.

Adversarial Causal Augmentation for Graph Covariate Shift

“…4.1 EXPERIMENTAL SETTINGS Datasets. We use graph OOD datasets (Gui et al, 2022) and OGB datasets (Hu et al, 2020), which include Motif, CMNIST, Molbbbp and Molhiv. Following Gui et al (2022), we adopt the base, color, size and scaffold data splitting to create various covariate shifts.…”

“…We use graph OOD datasets (Gui et al, 2022) and OGB datasets (Hu et al, 2020), which include Motif, CMNIST, Molbbbp and Molhiv. Following Gui et al (2022), we adopt the base, color, size and scaffold data splitting to create various covariate shifts. The details of the datasets, metrics and implementation details of AdvCA are provided in Appendix A.2 and A.3.…”

“…Secondly, if there exist environmental features in the test distribution that the model has not seen during training, it will also result in a large performance drop. This unseen distribution shift is well known as covariate shift in literature (Gui et al, 2022). Now we give a formal definition of the covariate shift on graphs.…”

“…Datasets. In this paper, we conduct experiments on graph OOD datasets (Gui et al, 2022) and OGB datasets (Hu et al, 2020), which include Motif, CMNIST, Molbbbp and Molhiv. We follow Gui et al (2022) to create various covariate shifts, according to base, color, size and scaffold splitting.…”

“…Following prior studies (Arjovsky et al, 2019;Wu et al, 2022b), we assume causal features are stable and invariant across distributions, in contrast to the environmental features. Correlation shift denotes that environments and labels establish inconsistent statistical correlations in training and test data; whereas, covariate shift means that the environmental features in test data are unseen in training data (Ye et al, 2022;Wiles et al, 2022;Gui et al, 2022). For example, in Figure 1, the environmental features ladder and tree are different in training and test data, which forms the covariate shift (↔).…”