Meta-Knowledge Transfer for Inductive Knowledge Graph Embedding

Chen, Mingyang; Zhang, Wen; Zhu, Yushan; Zhou, Hongting; Yuan, Zonggang; Xu, Changliang; Chen, Huajun

doi:10.1145/3477495.3531757

Cited by 46 publications

(13 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As a representative work of spatial methods, GCN [17] further simplifies graph convolution in the spectral domain by using first-order approximation, which enables graph convolution operations to be carried out in the spatial domain and greatly improves the computational efficiency of graph convolution models. Moreover, to speed up the training of graph neural networks, GNNs Graph-based Knowledge Distillation DKD methods Output layer DKWISL [18] KTG [19] DGCN [20] SPG [21] GCLN [22] Middle layer IEP [23] HKD [24] MHGD [25] IRG [26] DOD [27] HKDIFM [28] KDExplainer [29] TDD [30] DualDE [31] Constructed graph CAG [32] GKD [33] MorsE [34] BAF [35] LAD [36] GD [37] GCMT [38] GraSSNet [39] LSN [40] IntRA-KD [41] RKD [42] CC [43] SPKD [44] KCAN [45] GKD methods…”

Section: Graph Neural Networkmentioning

confidence: 99%

“…GKD [33] explores Frobenius to minimize the distribution difference between teachers and students and compresses the student model. MorsE [34] employs L 2 to transfer meta-knowledge to improve the student model in the link prediction and questionanswering system tasks.…”

Section: Graph-based Knowledge Distillation For Deep Neural Networkmentioning

confidence: 99%

“…Additionally, some related work has been proposed. Pan et al [21] argue that the previous video description model has not clearly modeled the interaction between objects and propose a novel spatiotemporal graph network that [23] KL, L1 Multi-task learning Transfer learning, image classification HKD [24] InfoCE Knowledge distillation Image classification, knowledge transfer CAG [32] KL Graph inference Visual dialogue DKWISL [18] KL Natural language processing Relation extraction KTG [19] KL Collaborative learning Image recognition MHGD [25] KL Multi-task learning Image recognition IRG [26] Hit Knowledge distillation Image recognition DGCN [20] KL Collaborative filtering Item recommendations GKD [33] Frobenius Model compression Image classification SPG [21] KL Natural language processing Video captioning MorsE [34] L2 Meta-knowledge transfer Link prediction, question answering system GCLN [22] L2 Image semantic segmentation Vision robot self-positioning DOD [27] KL Object detection Object Detectors BAF [35] EMD Model compression Video classification LAD [36] BELU Natural language processing Machine translation GD [37] Cosine Multimodal video Motion detection, action classification GCMT [38] CE Unsupervised domain adaptation Person re-identification GraSSNet [39] MSE Knowledge transfer Saliency prediction LSN [40] KL, MSE Model compression Node classification IntRA-KD [41] MSE Model compression Road marking segmentation RKD [42] Euclidean,Huber Knowledge distillation Image classification, few-Shot Learning CC [43] KL, MSE Knowledge distillation Image classification, person re-identification SPKD [44] Frobenius Knowledge distillation Image classification, transfer learning HKDIFM [28] KL Knowledge distillation Image classification KDExplainer [29] CE, KL Interpretability Image classification TDD [30] CE, KL Interpretability Image classification DualDE [31] JSD Knowledge distillation Node classification, link prediction KCAN [45] BPR Knowledge graph Top-K Recommendation, TR Prediction explicitly uses sp...…”

Section: Output Layer Knowledgementioning

confidence: 99%

“…Tu et al [45] study how to introduce external knowledge graph structure information into the recommendation network and distill the knowledge graph by spreading personalized information on the sampling subgraph through local conditional attention. Chen et al [34] perform a new task meta-knowledge transfer method MorsE for knowledge graph tasks (KGs), achieving excellent performance in link prediction and question-answering systems. Guo et al [32] utilize a fine-grained context-aware graph (CAG) distillation scheme for visual dialogue to solve the noise problem in visual dialogue tasks.…”

Section: Constructed Graph Knowledgementioning

confidence: 99%

See 3 more Smart Citations

Graph-based Knowledge Distillation: A survey and experimental evaluation

Liu¹,

Tongya²,

Zhang³

et al. 2023

Preprint

View full text Add to dashboard Cite

Graph data, such as citation networks, social networks, and transportation networks, are prevalent in the real world. Graph neural networks (GNNs) have gained widespread attention for their robust expressiveness and exceptional performance in various graph analysis applications. However, the efficacy of GNNs is heavily reliant on sufficient data labels and complex network models, with the former being challenging to obtain and the latter requiring expensive computational resources. To address the labeled data scarcity and high complexity of GNNs, Knowledge Distillation (KD) has been introduced to enhance existing GNNs. This technique involves transferring the soft-label supervision of the large teacher model to the small student model while maintaining prediction performance. Transferring the KD technique to graph data and graph-based knowledge is a major challenge. This survey offers a comprehensive overview of Graph-based Knowledge Distillation methods, systematically categorizing and summarizing them while discussing their limitations and future directions. This paper first introduces the background of graph and KD. It then provides a comprehensive summary of three types of Graph-based Knowledge Distillation methods, namely Graph-based Knowledge Distillation for deep neural networks (DKD), Graph-based Knowledge Distillation for GNNs (GKD), and Self-Knowledge Distillation based Graph-based Knowledge Distillation (SKD). Each type of method is further divided into knowledge distillation methods based on the output layer, middle layer, and constructed graph. Subsequently, various graph-based knowledge distillation algorithms' ideas are analyzed and compared, concluding with the advantages and disadvantages of each algorithm supported by experimental results. In addition, the applications of graph-based knowledge distillation in computer vision, natural language processing, recommendation systems, and other fields are listed. Finally, the development of graph-based knowledge distillation is summarized and prospectively discussed. We have also released related resources at https://github.com/liujing1023/Graph-based-Knowledge-Distillation.

show abstract

Section: Graph Neural Networkmentioning

confidence: 99%

Section: Graph-based Knowledge Distillation For Deep Neural Networkmentioning

confidence: 99%

Section: Output Layer Knowledgementioning

confidence: 99%

Section: Constructed Graph Knowledgementioning

confidence: 99%

See 2 more Smart Citations

Graph-based Knowledge Distillation: A survey and experimental evaluation

Liu¹,

Tongya²,

Zhang³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…For the further study on RCGNN, we are interested in continuing working on the combination of RL and GNN optimization, and learning how to better capture the features from the raw data during the representation learning process. In order to improve the model, transfer learning [22] and meta learning can be used [6]. We will keep enhancing the differentiation stability of the model and finally try to modify our model using dynamic and temporal graphs.…”

Section: Table II and Tablementioning

confidence: 99%

Reinforced Contrastive Graph Neural Networks (RCGNN) for Anomaly Detection

Sun

Jeon

et al. 2022

2022 IEEE International Performance, Computing, and Communications Conference (IPCCC)

View full text Add to dashboard Cite

Despite the recent state-of-the-art performance of Deep Learning (DL), imbalanced graph-structured data remains an open challenge in social science, traffic networks, and biomedical informatics. Recently, a surge in research on Representation Learning has significantly improved the performance of DL algorithms on imbalanced non-graph-structured data. In addition, Graph Neural Networks (GNNs) already in widespread use for representing graph-structured data in DL models with more advanced techniques in neural message-passing and deep graph embedding. However, most existing works are based on assumptions that oversimplify the complexity of real-world problems. In this paper, we propose Reinforced Contrastive GNNs (RCGNN), a novel graph representation learning model for anomaly detection with multi-relational graph-structured data. The proposed model produces a neighbor selection with Reinforcement Learning (RL) based on the similarity of neighborhoods in multi-relational structured graphs. In addition, the graph representation is learned by an adaptive AutoEncoder (AE) with Triplet Loss (TL) in Contrastive Learning. By aggregating the nodes with the highest similarities in their features and the importance of each node, our model is able to construct the multi-relational graphs by keeping the complexity of the graph structure as well as the relation-dependency representations. Experiments on multiple benchmark data sets demonstrate the advantage of RCGNN in learning better representations for multi-relational graphs. Furthermore, compared to other GNN models, our model shows better performance in accuracy, F1, and PR AUC scores.

show abstract