SAIL: Self-Augmented Graph Contrastive Learning

Yu, Lu; Pei, Shichao; Ding, Lizhong; Zhou, Jun; Li, Longfei; Zhang, Chuxu; Zhang, Xiangliang

doi:10.1609/aaai.v36i8.20875

Cited by 22 publications

(9 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This way, it alleviates the impact of false-negative samples. SAIL [29] utilized self-distillation to maintain distribution consistency between low-layer node embeddings and high-layer node features and alleviate the problem of smoothness. The idea behind AFGRL [30] is that augmentation on graphs is difficult to design.…”

Section: Deep Contrastive Graph Clusteringmentioning

confidence: 99%

Graph Clustering with High-Order Contrastive Learning

Li,

Zhu,

Wang

et al. 2023

Entropy

View full text Add to dashboard Cite

Graph clustering is a fundamental and challenging task in unsupervised learning. It has achieved great progress due to contrastive learning. However, we find that there are two problems that need to be addressed: (1) The augmentations in most graph contrastive clustering methods are manual, which can result in semantic drift. (2) Contrastive learning is usually implemented on the feature level, ignoring the structure level, which can lead to sub-optimal performance. In this work, we propose a method termed Graph Clustering with High-Order Contrastive Learning (GCHCL) to solve these problems. First, we construct two views by Laplacian smoothing raw features with different normalizations and design a structure alignment loss to force these two views to be mapped into the same space. Second, we build a contrastive similarity matrix with two structure-based similarity matrices and force it to align with an identity matrix. In this way, our designed contrastive learning encompasses a larger neighborhood, enabling our model to learn clustering-friendly embeddings without the need for an extra clustering module. In addition, our model can be trained on a large dataset. Extensive experiments on five datasets validate the effectiveness of our model. For example, compared to the second-best baselines on four small and medium datasets, our model achieved an average improvement of 3% in accuracy. For the largest dataset, our model achieved an accuracy score of 81.92%, whereas the compared baselines encountered out-of-memory issues.

show abstract

Section: Deep Contrastive Graph Clusteringmentioning

confidence: 99%

Graph Clustering with High-Order Contrastive Learning

Li,

Zhu,

Wang

et al. 2023

Entropy

View full text Add to dashboard Cite

show abstract

“…The differences between each SKD method are detailed in TABLE 4. For instance, SAIL [78], GNN-SD [79], and SDSS [80] use KL divergence for output layer, middle layer, and constructed graph knowledge, respectively. LinkDist [76] applies GKD using MSE distance for node classification and model compression.…”

Section: Self-knowledge Distillation Based Graph-based Knowledge Dist...mentioning

confidence: 99%

“…Therefore, researchers try to explore the rich information contained in the convolutional layer of the GNNs, hoping to mine the knowledge with more ability to express node features. The representative methods include GNN-SD [79] and SAIL [78].…”

Section: Middle Layer Knowledgementioning

confidence: 99%

“…Another representative work is the SAIL framework presented by Yu et al [78] in 2022. SAIL contains two complementary Self-Knowledge Distillation distillation modules, namely Intra-distill and Inter-distillation modules, which iteratively use the smoothing node feature of the GNN intermediate layer to correct the GNN shallow representation through Intra-distill and Inter-distill.…”

Section: Middle Layer Knowledgementioning

confidence: 99%

See 1 more Smart Citation

Graph-based Knowledge Distillation: A survey and experimental evaluation

Liu¹,

Tongya²,

Zhang³

et al. 2023

Preprint

View full text Add to dashboard Cite

Graph data, such as citation networks, social networks, and transportation networks, are prevalent in the real world. Graph neural networks (GNNs) have gained widespread attention for their robust expressiveness and exceptional performance in various graph analysis applications. However, the efficacy of GNNs is heavily reliant on sufficient data labels and complex network models, with the former being challenging to obtain and the latter requiring expensive computational resources. To address the labeled data scarcity and high complexity of GNNs, Knowledge Distillation (KD) has been introduced to enhance existing GNNs. This technique involves transferring the soft-label supervision of the large teacher model to the small student model while maintaining prediction performance. Transferring the KD technique to graph data and graph-based knowledge is a major challenge. This survey offers a comprehensive overview of Graph-based Knowledge Distillation methods, systematically categorizing and summarizing them while discussing their limitations and future directions. This paper first introduces the background of graph and KD. It then provides a comprehensive summary of three types of Graph-based Knowledge Distillation methods, namely Graph-based Knowledge Distillation for deep neural networks (DKD), Graph-based Knowledge Distillation for GNNs (GKD), and Self-Knowledge Distillation based Graph-based Knowledge Distillation (SKD). Each type of method is further divided into knowledge distillation methods based on the output layer, middle layer, and constructed graph. Subsequently, various graph-based knowledge distillation algorithms' ideas are analyzed and compared, concluding with the advantages and disadvantages of each algorithm supported by experimental results. In addition, the applications of graph-based knowledge distillation in computer vision, natural language processing, recommendation systems, and other fields are listed. Finally, the development of graph-based knowledge distillation is summarized and prospectively discussed. We have also released related resources at https://github.com/liujing1023/Graph-based-Knowledge-Distillation.

show abstract

“…One needs to ensure that paired features are similar both in modeling capacity and relevance to the output. Most research on feature-based distillation on graphs has so far focused on models that only have one type of (scalar) features in single-output classification tasks [26,27,28,29], thereby reducing the problem to the selection of layers to pair across the student and the teacher. This is often further simplified by utilizing models of the same architecture.…”

Section: Knowledge Distillation In Molecular Gnnsmentioning

confidence: 99%