Cold Brew: Distilling Graph Node Representations with Incomplete or Missing Neighborhoods

Abstract

Graph Neural Networks (GNNs) have achieved state of the art performance in node classification, regression, and recommendation tasks. GNNs work well when high-quality and rich connectivity structure is available. However, this requirement is not satisfied in many real world graphs where the node degrees have power-law distributions as many nodes have either fewer or noisy connections. The extreme case of this situation is a node may have no neighbors at all, called Strict Cold Start (SCS) scenario. This forces the prediction models to rely completely on the node's input features. We propose Cold Brew to address the SCS and noisy neighbor setting compared to pointwise and other graph-based models via a distillation approach. We introduce feature-contribution ratio (FCR), a metric to study the viability of using inductive GNNs to solve the SCS problem and to select the best architecture for SCS generalization. We experimentally show FCR disentangles the contributions of various components of graph datasets and demonstrate the superior performance of Cold Brew on several public benchmarks and proprietary e-commerce datasets. The source code for our approach is available at: https: //github.com/amazon-research/gnn-tail-generalization.

Graph-based Knowledge Distillation: A survey and experimental evaluation

“…GFKD [46] RDD [47] GKD [48] GLNN [49] Distill2Vec [50] MT-GCN [51] TinyGNN [52] GLocalKD [53] SCR [54] ROD [55] EGNN [56] Middle layer LWC-KD [57] MustaD [58] EGAD [59] AGNN [60] Cold Brew [61] PGD [62] OAD [63] CKD [64] BGNN [65] EGSC [66] HSKDM [67] Constructed graph GRL [68] GFL [69] HGKT [70] CPF [71] LSP [16] scGCN [72] MetaHG [73] G-CRD [74] HIRE [75] SKD methods…”

“…Output layer-based methods using KL divergence are GFKD [46], GLNN [49], Distill2Vec [50], and MT-GCN [51]; middle layer-based methods using KL divergence are MustaD [58], OAD [63], BGNN [65], and HSKDM [67]; constructed graph-based methods using KL divergence are CPF [71], LSP [16], MetaHG [73], and HIRE [75]. Similarly, output layer-based methods using MSE are RDD [47] and SCR [54]; middle layer-based methods using MSE are LWC-KD [57], AGNN [60], Cold Brew [61], and EGSC [66]. Finally, HIRE [75] uses the constructed graph knowledge of a relational metric types to measure the differences between teacher and student models.…”

“…The other category is the middle layer knowledge of GKD for other graph analysis tasks such as recommendation systems [57,61,62]. Zheng et al [61] develop knowledge distillation technology to embed teacher nodes on lowdimensional manifolds by using graph structures and require students to learn the mapping from node features to the manifold, generalizing GNN to cold start research problems in recommender systems.…”

“…The other category is the middle layer knowledge of GKD for other graph analysis tasks such as recommendation systems [57,61,62]. Zheng et al [61] develop knowledge distillation technology to embed teacher nodes on lowdimensional manifolds by using graph structures and require students to learn the mapping from node features to the manifold, generalizing GNN to cold start research problems in recommender systems. In the same year, Wang et al [57] combine contrast learning and distillation learning and propose a layer-by-layer comparative distillation framework named LWC-KD for the recommended scenario of incremental learning.…”

Old can be Gold: Better Gradient Flow can Make Vanilla-GCNs Great Again

“…Graphs are omnipresent and Graphs convolutional networks (GCNs) [1] and their variants [2,3,4,5,6,7,8,9,10] are a powerful family of neural networks which can learn from graph-structured data. GCNs have been enormously successful in numerous real-world applications such as recommendation systems [11,12], social and academic networks [13,11,5,4,14], modeling proteins for drug discovery [15,16,12], computer vision [17,18], etc.…”