Graph Neural Networks with Learnable Structural and Positional Representations

Abstract

Graph neural networks (GNNs) have become the standard learning architectures for graphs. GNNs have been applied to numerous domains ranging from quantum chemistry, recommender systems to knowledge graphs and natural language processing. A major issue with arbitrary graphs is the absence of canonical positional information of nodes, which decreases the representation power of GNNs to distinguish e.g. isomorphic nodes and other graph symmetries. An approach to tackle this issue is to introduce Positional Encoding (PE) of nodes, and inject it into the input layer, like in Transformers. Possible graph PE are Laplacian eigenvectors. In this work, we propose to decouple structural and positional representations to make easy for the network to learn these two essential properties. We introduce a novel generic architecture which we call LSPE (Learnable Structural and Positional Encodings). We investigate several sparse and fully-connected (Transformer-like) GNNs, and observe a performance increase for molecular datasets, from 2.87% up to 64.14% when considering learnable PE for both GNN classes. 1

Structure and position-aware graph neural network for airway labeling

“…In Transformers [27], positional encodings are concatenated with the word embeddings as the input to the learning process without being involved in the layer-wise update, referred to as non-learnable positional encodings (NLPE). Similar to [30], in our work, we iteratively update positional encodings using a dedicated GNN such that positional encodings can be adjusted to the graph structure at hand. The result using learnable position encodings is denoted as PE.…”

“…Assuming the CNN predictions are mostly accurate (results in Table II), the correctly-predicted branches can be used to provide canonical positional encodings because their locations are consistently defined (up, down, left, and right) according to the airway tree anatomy. Many existing positional encoding methods, i.e., Laplacian eigenvectors [29], randomwalks encodings [30], and encodings using random anchors [16] provide non-canonical positional encodings because these methods operate on arbitrary graphs. By adding leaf branches, the anchors distribute evenly in terms of the depth of the tree, preventing selecting anchors from only upper side of the tree.…”

LogGD:Detecting Anomalies from System Logs by Graph Neural Networks

“…GNNs typically employ a neighborhood aggregation strategy [25], [26], where the representations of nodes in the graph are iteratively updated by aggregating representations of their neighbors. Ultimately, a node's high-level representation captures structural attributes within its L-hop network neighborhood.…”

“…It plays a crucial role in improving model effectiveness. Studies [25], [47] have shown that PE representing the structural attributes of graphs is also essential for prediction tasks on graphs. However, finding such positional encodings on graphs for nodes is challenging due to the invariance of graphs to node permutation.…”

“…Regarding spectral PE, it uses Laplacian Eigenvectors as a meaningful local coordinate system to conserve the global graph structure. However, spectral PE suffers from sign ambiguity, which requires random sign flipping during training for the network to learn the invariance [25]. Diffusion-based PE, such as [25], [52], is based on the diffusion process, such as Random Walk and PageRank.…”