Hierarchical Transformer for Scalable Graph Learning

Abstract

Graph Transformer is gaining increasing attention in the field of machine learning and has demonstrated state-of-the-art performance on benchmarks for graph representation learning. However, as current implementations of Graph Transformer primarily focus on learning representations of small-scale graphs, the quadratic complexity of the global self-attention mechanism presents a challenge for full-batch training when applied to larger graphs. Additionally, conventional sampling-based methods fail to capture necessary high-level contextual information, resulting in a significant loss of performance. In this paper, we introduce the Hierarchical Scalable Graph Transformer (HSGT) as a solution to these challenges. HSGT successfully scales the Transformer architecture to node representation learning tasks on large-scale graphs, while maintaining high performance. By utilizing graph hierarchies constructed through coarsening techniques, HSGT efficiently updates and stores multi-scale information in node embeddings at different levels. Together with sampling-based training methods, HSGT effectively captures and aggregates multi-level information on the hierarchical graph using only Transformer blocks. Empirical evaluations demonstrate that HSGT achieves state-of-the-art performance on large-scale benchmarks with graphs containing millions of nodes with high efficiency.

Dual-channel graph contrastive learning for self-supervised graph-level representation learning

“…Observe that, empirically, we mainly focus on a maximal hierarchy level of K = 1. This is in line with related works (Zhang et al 2022;Zhu et al 2023) and showed good performance in our evaluation.…”

“…Furthermore, as the K increases, there is no significant improvement in the results. This might explain why we chose K = 1 in line with related works (Zhang et al 2022;Zhu et al 2023).…”

“…Essentially, their ANS-GT groups the original nodes into so-called super-nodes and enables the former to interact with sampled super nodes. Similarly, HSGT (Zhu et al 2023) aims at aggregating multi-level graph information and employs graph hierarchical structure to construct intra-level and inter-level transformer blocks. In this framework, the intra-level block facilitates the exchange and transformation of information within the local context of each node, while the inter-level block adaptively coalesces every substructure present.…”

Deep learning for graph structured data

“…Clustering based GTs. Orthogonally, several recent works use hierarchical clustering or partitioning to perform global attention on the coarsened or super nodes [189,190]. However, the coarsening step remains intractable for very large Local Inductive Bias: A node's local neighborhood connectivity presents a rich source of information that is essential to utilize even in a large graph setting.…”