Graph Self-Supervised Learning: A Survey

Liu, Yixin; Jin, Ming; Pan, Shirui; Zhou, Chuan; Zheng, Yu; Xia, Feng; Yu, Philip S.

doi:10.1109/tkde.2022.3172903

Cited by 219 publications

(81 citation statements)

References 127 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Pre-training of GNNs. Various recent works have formulated methods for pre-training using graph data [39,27,75,31], rather than 3D point clouds of atom nuclei as in this paper. Approaches based on contrastive methods rely on learning representations by contrasting different views of the input graph [64,69,79,37], or bootstrapping [65].…”

Section: Related Workmentioning

confidence: 99%

Pre-training via Denoising for Molecular Property Prediction

Zaidi¹,

Schaarschmidt²,

Martens³

et al. 2022

Preprint

View full text Add to dashboard Cite

Many important problems involving molecular property prediction from 3D structures have limited data, posing a generalization challenge for neural networks. In this paper, we describe a pre-training technique that utilizes large datasets of 3D molecular structures at equilibrium to learn meaningful representations for downstream tasks. Inspired by recent advances in noise regularization, our pre-training objective is based on denoising. Relying on the well-known link between denoising autoencoders and score-matching, we also show that the objective corresponds to learning a molecular force field -arising from approximating the physical state distribution with a mixture of Gaussians -directly from equilibrium structures. Our experiments demonstrate that using this pre-training objective significantly improves performance on multiple benchmarks, achieving a new state-of-the-art on the majority of targets in the widely used QM9 dataset. Our analysis then provides practical insights into the effects of different factors -dataset sizes, model size and architecture, and the choice of upstream and downstream datasets -on pre-training.

show abstract

Section: Related Workmentioning

confidence: 99%

Pre-training via Denoising for Molecular Property Prediction

Zaidi¹,

Schaarschmidt²,

Martens³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Graph Self-Supervised Learning: In graphs, recent works have explored several paradigms for self-supervised learning: see [62] for an up-to-date survey. Graph pre-text tasks are often reminiscent of image in-painting tasks [63], and seek to complete masked graphs and/or node features ( [64,13]).…”

Section: G Related Workmentioning

confidence: 99%

Analyzing Data-Centric Properties for Graph Contrastive Learning

Trivedi¹,

Lubana²,

Heimann³

et al. 2022

Preprint

View full text Add to dashboard Cite

Recent analyses of self-supervised learning (SSL) find the following data-centric properties to be critical for learning good representations: invariance to taskirrelevant semantics, separability of classes in some latent space, and recoverability of labels from augmented samples. However, given their discrete, non-Euclidean nature, graph datasets and graph SSL methods are unlikely to satisfy these properties. This raises the question: how do graph SSL methods, such as contrastive learning (CL), work well? To systematically probe this question, we perform a generalization analysis for CL when using generic graph augmentations (GGAs), with a focus on data-centric properties. Our analysis yields formal insights into the limitations of GGAs and the necessity of task-relevant augmentations. As we empirically show, GGAs do not induce task-relevant invariances on common benchmark datasets, leading to only marginal gains over naive, untrained baselines. Our theory motivates a synthetic data generation process that enables control over task-relevant information and boasts pre-defined optimal augmentations. This flexible benchmark helps us identify yet unrecognized limitations in advanced augmentation techniques (e.g., automated methods). Overall, our work rigorously contextualizes, both empirically and theoretically, the effects of data-centric properties on augmentation strategies and learning paradigms for graph SSL.

show abstract

“…Graph Neural Networks. Over the last decade, hundreds of GNN methods, libraries, and software have been developed for semisupervised, unsupervised, and self-supervised learning on graphs [1,[13][14][15][16][17][18]. Their success in graph learning tasks and their limitations such as over-smoothing and neighborhood explosion problems are also well-documented in the literature [19][20][21].…”

Section: Related Workmentioning

confidence: 99%

Triple Sparsification of Graph Convolutional Networks without Sacrificing the Accuracy

Rahman¹,

Azad²

2022

Preprint

View full text Add to dashboard Cite

Graph Neural Networks (GNNs) are widely used to perform different machine learning tasks on graphs. As the size of the graphs grows, and the GNNs get deeper, training and inference time become costly in addition to the memory requirement. Thus, without sacrificing accuracy, graph sparsification, or model compression becomes a viable approach for graph learning tasks. A few existing techniques only study the sparsification of graphs and GNN models. In this paper, we develop a SparseGCN pipeline to study all possible sparsification in GNN. We provide a theoretical analysis and empirically show that it can add up to 11.6% additional sparsity to the embedding matrix without sacrificing the accuracy of the commonly used benchmark graph datasets.

show abstract

Graph Self-Supervised Learning: A Survey

Cited by 219 publications

References 127 publications

Pre-training via Denoising for Molecular Property Prediction

Pre-training via Denoising for Molecular Property Prediction

Analyzing Data-Centric Properties for Graph Contrastive Learning

Triple Sparsification of Graph Convolutional Networks without Sacrificing the Accuracy

Contact Info

Product

Resources

About