Accelerating Large-Scale Heterogeneous Interaction Graph Embedding Learning via Importance Sampling

Abstract

In real-world problems, heterogeneous entities are often related to each other through multiple interactions, forming a Heterogeneous Interaction Graph (HIG). While modeling HIGs to deal with fundamental tasks, graph neural networks present an attractive opportunity that can make full use of the heterogeneity and rich semantic information by aggregating and propagating information from different types of neighborhoods. However, learning on such complex graphs, often with millions or billions of nodes, edges, and various attributes, could suffer from expensive time cost and high memory consumption. In this article, we attempt to accelerate representation learning on large-scale HIGs by adopting the importance sampling of heterogeneous neighborhoods in a batch-wise manner, which naturally fits with most batch-based optimizations. Distinct from traditional homogeneous strategies neglecting semantic types of nodes and edges, to handle the rich heterogeneous semantics within HIGs, we devise both type-dependent and type-fusion samplers where the former respectively samples neighborhoods of each type and the latter jointly samples from candidates of all types. Furthermore, to overcome the imbalance between the down-sampled and the original information, we respectively propose heterogeneous estimators including the self-normalized and the adaptive estimators to improve the robustness of our sampling strategies. Finally, we evaluate the performance of our models for node classification and link prediction on five real-world datasets, respectively. The empirical results demonstrate that our approach performs significantly better than other state-of-the-art alternatives, and is able to reduce the number of edges in computation by up to 93%, the memory cost by up to 92% and the time cost by up to 86%.

GIPA: A General Information Propagation Algorithm for Graph Learning

“…and how much contribution each neighboring node makes to the aggregated embedding?. For the former question, neighborhood sampling [10,42,4,13,46,14] is proposed for large dense or power-law graphs. For the latter, neighbor importance estimation is used to attach different weights to different neighboring nodes during feature propagation.…”

“…For the latter, neighbor importance estimation is used to attach different weights to different neighboring nodes during feature propagation. Importance sampling [4,46,14] and attention [34,21,37,43,12] are two popular techniques. Importance sampling is a special case of neighborhood sampling, where the importance weight of a neighboring node is drawn from a distribution over nodes.…”

“…Importance sampling is a special case of neighborhood sampling, where the importance weight of a neighboring node is drawn from a distribution over nodes. This distribution can be derived from normalized Laplacian matrices [4,46] or jointly learned with GNNs [14]. With this distribution, at each step a subset of neighbors is sampled, and aggregated with the importance weights.…”