Pairwise Half-graph Discrimination: A Simple Graph-level Self-supervised Strategy for Pre-training Graph Neural Networks

Abstract

Self-supervised learning has gradually emerged as a powerful technique for graph representation learning. However, transferable, generalizable, and robust representation learning on graph data still remains a challenge for pre-training graph neural networks. In this paper, we propose a simple and effective self-supervised pre-training strategy, named Pairwise Half-graph Discrimination (PHD), that explicitly pre-trains a graph neural network at graph-level. PHD is designed as a simple binary classification task to discriminate whether two half-graphs come from the same source. Experiments demonstrate that the PHD is an effective pre-training strategy that offers comparable or superior performance on 13 graph classification tasks compared with state-of-the-art strategies, and achieves notable improvements when combined with node-level strategies. Moreover, the visualization of learned representation revealed that PHD strategy indeed empowers the model to learn graph-level knowledge like the molecular scaffold. These results have established PHD as a powerful and effective self-supervised learning strategy in graph-level representation learning.

Improving drug response prediction via integrating gene relationships with deep learning

“…where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $\mathcal{N}_{i}$\end{document} represents the neighbors of node \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $i$\end{document} , \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $e_{i j}$\end{document} denotes the edge between the node \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $i$\end{document} and node \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $j$\end{document} , vertex update function U is a gated recurrent unit network [ 43 ], \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $h_{i}^{t-1}$\end{document} is the hidden state of U and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $h_{i}^{0}$\end{document} is the initial atom representation \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $x_{i}^{0}$\end{document} . The MolGNet is pre-trained with both Pairwise Half-graph Discrimination [ 44 ], which is designed to discriminate whether two half-graphs are of the same source, and graph attribute masking strategy.…”

Neural Graph Matching for Pre-training Graph Neural Networks

“…The first is called node-level tasks, which aims at predicting localized properties utilizing node representations, such as graph structure reconstruction [3,6,7,15,11], localized attribute prediction [6,7] and node representation recovery [4]. Another is called graph-level tasks, which defines globalized optimization goal for the entire graph, such as graph property prediction [6,12] and mutual information maximization [21,28,17,15,27]. Our proposed framework differs from the above approaches in the following two aspects: learning node-and graph-level graph semantics in one single pre-training task and adaptive graph representations.…”

Artificial intelligence in recommender systems

“…In particular, general graph SSL work [You et al 2020;Xie et al 2022c] has considered molecule representation learning as an important use case. In contrast, other studies have developed SSL methods specifically for molecular graphs, which allows for the integration of domain knowledge such as functional groups (motifs) co-occurrence [Hu et al 2019b;Rong et al 2020;Li et al 2021f], atom-bond associations [Rong et al 2020], and reaction context ].…”