Rethinking Graph Transformers with Spectral Attention

Abstract

In recent years, the Transformer architecture has proven to be very successful in sequence processing, but its application to other data structures, such as graphs, has remained limited due to the difficulty of properly defining positions. Here, we present the Spectral Attention Network (SAN), which uses a learned positional encoding (LPE) that can take advantage of the full Laplacian spectrum to learn the position of each node in a given graph. This LPE is then added to the node features of the graph and passed to a fully-connected Transformer. By leveraging the full spectrum of the Laplacian, our model is theoretically powerful in distinguishing graphs, and can better detect similar sub-structures from their resonance. Further, by fully connecting the graph, the Transformer does not suffer from over-squashing, an information bottleneck of most GNNs, and enables better modeling of physical phenomenons such as heat transfer and electric interaction. When tested empirically on a set of 4 standard datasets, our model performs on par or better than state-of-theart GNNs, and outperforms any attention-based model by a wide margin, becoming the first fully-connected architecture to perform well on graph benchmarks. * Equal contribution.Preprint. Under review.

GRPE: Relative Positional Encoding for Graph Transformer

“…There are many attempts of leveraging transformer into the graph domain. Existing methods (Veličković et al, 2017;Kreuzer et al, 2021;Zhang et al, 2020;Ying et al, 2021) have improved the transformer architecture to fit the graph input by improving the attention map or replacing the positional embedding to fit the graph, or both, which are more related to our GSA. Methods like Graph Attention Networks (GAT) (Veličković et al, 2017) and Graph Transformer (GT) constrain the self-attention mechanism to neighboring nodes.…”

“…They surpass GNN baseline methods on the graph representation task. Spectral Attention Network (SAN) (Kreuzer et al, 2021) employs a learned positional encoding (LPE) of Laplacian spectrum to learn the position of nodes in a graph. Graph-BERT (Zhang et al, 2020) uses several types of relative positional encodings to embed the information about the edges within a subgraph.…”

“…However, learning a graph representation with a Transformer model is particularly challenging due to the structural flexibility of a graph, because the Transformer was devised for sequential data, not for a graph. Recently, several algorithms have been proposed to leverage Transformers for learning graph representation Kreuzer et al, 2021;Ying et al, 2021).…”

“…For instance, Ying et al (2021) encodes graph information on the attention map by adding bias terms and encodes graph information on the input with centrality encoding. Meanwhile, Dwivedi & Bresson (2020); Kreuzer et al (2021) replace positional encoding into graph Laplacian (Belkin & Niyogi, 2003). We examine possible improvements from previous methods and propose our novel self-attention module for learning graph representation on Transformer.…”

Accelerate Model Parallel Training by Using Efficient Graph Traversal Order in Device Placement

“…The GNN that are used by state-of-the-art device placement methods mostly follow the message-passing paradigm, which is known to have inherent limitations. For example, the expressiveness of such GNN is bounded by the Weisfeiler-Lehman isomorphism hierarchy [23]. Also, GNNs are known to suffer from over-squashing [24], where there is a distortion of information propagation between distant nodes.…”