Graph convolutions that can finally model local structure

Brossard, Rémy; Frigo, Oriel; Dehaene, David

doi:10.48550/arxiv.2011.15069

Cited by 14 publications

(30 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Although our method and DGN [4] all require eigendecomposition in the preprocessing step, non-spatial GN significantly outperforms DGN on ZINC and MolPCBA. GIN [75] 509,549 0.526˘0.051 GraphSage [23] 505,341 0.398˘0.002 GAT [66] 531,345 0.384˘0.007 GCN [35] 505,079 0.367˘0.011 GatedGCN-PE [8] 505,011 0.214˘0.006 MPNN (sum) [22] 480,805 0.145˘0.007 PNA [16] 387,155 0.142˘0.010 DGN [4] -0.168˘0.003 GSN [7] 523,201 0.101˘0.010 GT [19] 588,929 0.226˘0.014 SAN [39] 508, 577 0.139˘0.006 GraphormerSLIM [79] GCN [35] 0.56M 20.20˘0.24 GIN [75] 1.92M 22.66˘0.28 GCN-VN [35] 2.02M 24.24˘0.34 GIN-VN [75] 3.37M 27.03˘0.23 GCN-VN+FLAG [38] 2.02M 24.83˘0.37 GIN-VN+FLAG [38] 3.37M 28.34˘0.38 DeeperG-VN+FLAG [43] 5.55M 28.42˘0.43 PNA [16] 6.55M 28.38˘0.35 DGN [4] 6.73M 28.85˘0.30 GINE-VN [10] 6.15M 29.17˘0.15 GINE-APPNP [10] 6.15M 29.79˘0.30 PHC-GNN [40] 1.69M 29.47˘0.26…”

Section: Resultsmentioning

confidence: 99%

A New Perspective on the Effects of Spectrum in Graph Neural Networks

Yang¹,

Shen²,

Li³

et al. 2021

Preprint

View full text Add to dashboard Cite

From the original theoretically well-defined spectral graph convolution to the subsequent spatial bassed message-passing model, spatial locality (in vertex domain) acts as a fundamental principle of most graph neural networks (GNNs). In the spectral graph convolution, the filter is approximated by polynomials, where a k-order polynomial covers k-hop neighbors. In the message-passing, various definitions of neighbors used in aggregations are actually an extensive exploration of the spatial locality information. For learning node representations, the topological distance seems necessary since it characterizes the basic relations between nodes. However, for learning representations of the entire graphs, is it still necessary to hold? In this work, we show that such a principle is not necessary, it hinders most existing GNNs from efficiently encoding graph structures. By removing it, as well as the limitation of polynomial filters, the resulting new architecture significantly boosts performance on learning graph representations. We also study the effects of graph spectrum on signals and interpret various existing improvements as different spectrum smoothing techniques. It serves as a spatial understanding that quantitatively measures the effects of the spectrum to input signals in comparison to the well-known spectral understanding as high/low-pass filters. More importantly, it sheds the light on developing powerful graph representation models.Preprint. Under review.

show abstract

Section: Resultsmentioning

confidence: 99%

A New Perspective on the Effects of Spectrum in Graph Neural Networks

Yang¹,

Shen²,

Li³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…They achieve the state-of-the-art valid and test mean absolute error (MAE) on the official leaderboard 4 [20]. In addition, we compare to GIN's multi-hop variant [5], and 12-layer deep graph network DeeperGCN [29], which also show promising performance on other leaderboards. We further compare our Graphormer with the recent Transformer-based graph model GT [13].…”

Section: Ogb Large-scale Challengementioning

confidence: 93%

“…All models are trained on 8 NVIDIA V100 GPUS for about 2 days. [5,15] 13.2M 0.1248 0.1430 -DeeperGCN-VN [29,15] 25.5M 0.1059 0.1398 -GT [13] 0.6M 0.0944 0.1400 -GT-Wide [13] 83 [13] employs a hidden dimension of 64 to reduce the total number of parameters. For a fair comparison, we also report the result by enlarging the hidden dimension to 768, denoted by GT-Wide, which leads to a total number of parameters of 83.2M.…”

Section: Ogb Large-scale Challengementioning

confidence: 99%

“…Compared to the previous state-of-the-art GNN architecture, Graphormer noticeably surpasses GIN-VN by a large margin, e.g., 11.5% relative validate MAE decline. Noting that the test dataset is publicly unavailable, and the test MAE of Graphormer is evaluated by the OGB team on 5% of test data 5 . As stated in Section 3.3, we further find that the proposed Graphormer does not encounter the problem of over-smoothing, i.e., the train and validate error keep going down along with the growth of depth and width of models.…”

Section: Ogb Large-scale Challengementioning

confidence: 99%

See 1 more Smart Citation

Do Transformers Really Perform Bad for Graph Representation?

Ying¹,

Cai²,

Luo³

et al. 2021

Preprint

View full text Add to dashboard Cite

The Transformer architecture has become a dominant choice in many domains, such as natural language processing and computer vision. Yet, it has not achieved competitive performance on popular leaderboards of graph-level prediction compared to mainstream GNN variants. Therefore, it remains a mystery how Transformers could perform well for graph representation learning. In this paper, we solve this mystery by presenting Graphormer, which is built upon the standard Transformer architecture, and could attain excellent results on a broad range of graph representation learning tasks, especially on the recent OGB Large-Scale Challenge. Our key insight to utilizing Transformer in the graph is the necessity of effectively encoding the structural information of a graph into the model. To this end, we propose several simple yet effective structural encoding methods to help Graphormer better model graph-structured data. Besides, we mathematically characterize the expressive power of Graphormer and exhibit that with our ways of encoding the structural information of graphs, many popular GNN variants could be covered as the special cases of Graphormer. The code and models of Graphormer will be made publicly available at https://github.com/Microsoft/Graphormer.

show abstract

“…OGB. We use GNNs achieving top places on the OGB graph classification leaderboard 3 (at the time of submission) as the baselines, including GCN [12], GIN [27], DeeperGCN [67], Deep LRP [39], PNA [68], DGN [33], GINE [69], and PHC-GNN [70]. Note that those high-order GNNs [19][20][21]25] are not included here, because despite being theoretically more discriminative, these GNNs are not among the GNNs with the best empirical performance on modern large-scale graph benchmarks, and their O(n 3 ) complexity also raises a scalability issue.…”

Section: Modelsmentioning

confidence: 99%

Nested Graph Neural Networks

Zhang¹,

Li²

2021

Preprint

View full text Add to dashboard Cite

Graph neural network (GNN)'s success in graph classification is closely related to the Weisfeiler-Lehman (1-WL) algorithm. By iteratively aggregating neighboring node features to a center node, both 1-WL and GNN obtain a node representation that encodes a rooted subtree around the center node. These rooted subtree representations are then pooled into a single representation to represent the whole graph. However, rooted subtrees are of limited expressiveness to represent a nontree graph. To address it, we propose Nested Graph Neural Networks (NGNNs). NGNN represents a graph with rooted subgraphs instead of rooted subtrees, so that two graphs sharing many identical subgraphs (rather than subtrees) tend to have similar representations. The key is to make each node representation encode a subgraph around it more than a subtree. To achieve this, NGNN extracts a local subgraph around each node and applies a base GNN to each subgraph to learn a subgraph representation. The whole-graph representation is then obtained by pooling these subgraph representations. We provide a rigorous theoretical analysis showing that NGNN is strictly more powerful than 1-WL. In particular, we proved that NGNN can discriminate almost all r-regular graphs, where 1-WL always fails. Moreover, unlike other more powerful GNNs, NGNN only introduces a constantfactor higher time complexity than standard GNNs. NGNN is a plug-and-play framework that can be combined with various base GNNs. We test NGNN with different base GNNs on several benchmark datasets. NGNN uniformly improves their performance and shows highly competitive performance on all datasets.

show abstract

Graph convolutions that can finally model local structure

Cited by 14 publications

References 0 publications

A New Perspective on the Effects of Spectrum in Graph Neural Networks

A New Perspective on the Effects of Spectrum in Graph Neural Networks

Do Transformers Really Perform Bad for Graph Representation?

Nested Graph Neural Networks

Contact Info

Product

Resources

About