CAP: Co-Adversarial Perturbation on Weights and Features for Improving Generalization of Graph Neural Networks

Abstract

Despite the recent advances of graph neural networks (GNNs) in modeling graph data, the training of GNNs on large datasets is notoriously hard due to the overfitting. Adversarial training, which augments data with the worst-case adversarial examples, has been widely demonstrated to improve model's robustness against adversarial attacks and generalization ability. However, while the previous adversarial training generally focuses on protecting GNNs from spiteful attacks, it remains unclear how the adversarial training could improve the generalization abilities of GNNs in the graph analytics problem. In this paper, we investigate GNNs from the lens of weight and feature loss landscapes, i.e., the loss changes with respect to model weights and node features, respectively. We draw the conclusion that GNNs are prone to falling into sharp local minima in these two loss landscapes, where GNNs possess poor generalization performances. To tackle this problem, we construct the co-adversarial perturbation (CAP) optimization problem in terms of weights and features, and design the alternating adversarial perturbation algorithm to flatten the weight and feature loss landscapes alternately. Furthermore, we divide the training process into two stages: one conducting the standard cross-entropy minimization to ensure the quick convergence of GNN models, the other applying our alternating adversarial training to avoid falling into locally sharp minima. The extensive experiments demonstrate our CAP can generally improve the generalization performance of GNNs on a variety of benchmark graph datasets.

stAA: adversarial graph autoencoder for spatial clustering task of spatially resolved transcriptomics

“…These existing spatial clustering methods often encounter the problem of learning degenerate identity mappings, where the latent embedding space does not have meaningful structure. In addition, GNN can be prone to overfitting [ 21 ], which hinders the accurate identification of spatial domain boundaries.…”

Insights into the Discrepancy between Single Molecule Experiments*

“…However, GNNs may cause over-fitting, reducing their generalization ability [17]. Furthermore, node representations tend to become indistinguishable due to the message-passing mechanism aggregating information from neighbors which called over-smoothing.…”