Measuring and Relieving the Over-Smoothing Problem for Graph Neural Networks from the Topological View

Chen, Deli; Lin, Yankai; Li, Wei; Li, Peng; Zhou, Jie; Sun, Xu

doi:10.1609/aaai.v34i04.5747

Cited by 691 publications

(362 citation statements)

References 11 publications

Supporting

Mentioning

354

Contrasting

Unclassified

Order By: Relevance

“…(2) As shown in Figure 5g, it seems that the over-smoothing problem still exists in MADReg. Similar conclusions can be found in the original paper of MADReg [30]. Specifically, from Table 5 in [30], we can find that although MADReg can improve the performance of the standard GNNs, the performance of a GNN with the MADReg in a deep version is far away from the performance of that in a shallow version.…”

Section: Evaluation Of Rejection Mechanismsupporting

confidence: 87%

“…Normalizing the representations of nodes can prevent these representations from being too similar. MADReg [30] provides a topological view of the over-smoothing problem and brings a MADReg loss to avoid the representations of the distant nodes to be similar to the representations of the neighbor nodes. Experimental results (see Section 6.5) show MADReg can only relieve rather than prevent this problem.…”

Section: Related Workmentioning

confidence: 99%

“…In this way, PN can address the over-smoothing problem. MADReg [30] also focuses on the over-smoothing problem. It takes the gap of MAD values (MADGap) as a regularization term to avoid the representations of distant nodes to be similar.…”

Section: Baselinesmentioning

confidence: 99%

“…Evaluation with the GNNs, which also address the over-smoothing problem. Then, we evaluate the RM with the GNNs (i.e., JK [21], PN [29], and MADReg [30]), which also address the over-smoothing problem. (1) As seen in Figure 5e-f, both JK and PN can well solve the over-smoothing problem, as the performances do not drop sharply with the increasing of the layer number.…”

Section: Evaluation Of Rejection Mechanismmentioning

confidence: 99%

See 3 more Smart Citations

Graph Dilated Network with Rejection Mechanism

2020

View full text Add to dashboard Cite

Recently, graph neural networks (GNNs) have achieved great success in dealing with graph-based data. The basic idea of GNNs is iteratively aggregating the information from neighbors, which is a special form of Laplacian smoothing. However, most of GNNs fall into the over-smoothing problem, i.e., when the model goes deeper, the learned representations become indistinguishable. This reflects the inability of the current GNNs to explore the global graph structure. In this paper, we propose a novel graph neural network to address this problem. A rejection mechanism is designed to address the over-smoothing problem, and a dilated graph convolution kernel is presented to capture the high-level graph structure. A number of experimental results demonstrate that the proposed model outperforms the state-of-the-art GNNs, and can effectively overcome the over-smoothing problem.

show abstract

Section: Evaluation Of Rejection Mechanismsupporting

confidence: 87%

Section: Related Workmentioning

confidence: 99%

Section: Baselinesmentioning

confidence: 99%

Section: Evaluation Of Rejection Mechanismmentioning

confidence: 99%

See 2 more Smart Citations

Graph Dilated Network with Rejection Mechanism

2020

View full text Add to dashboard Cite

show abstract

“…While the idea of learning hierarchical graph representations makes sense, hierarchical GNNs do not consistently outperform flat GNNs [19]. In addition, with advanced techniques like jumping knowledge networks (JK-Net) [11] to address the over-smoothing problem of GNN layers [22], flat GNNs can go deeper and achieve better performance than hierarchical GNNs [12].…”

Section: Graph Pooling: Global Versus Hierarchicalmentioning

confidence: 99%

Second-Order Pooling for Graph Neural Networks

Wang

2023

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

Graph neural networks have achieved great success in learning node representations for graph tasks such as node classification and link prediction. Graph representation learning requires graph pooling to obtain graph representations from node representations. It is challenging to develop graph pooling methods due to the variable sizes and isomorphic structures of graphs. In this work, we propose to use second-order pooling as graph pooling, which naturally solves the above challenges. In addition, compared to existing graph pooling methods, second-order pooling is able to use information from all nodes and collect second-order statistics, making it more powerful. We show that direct use of second-order pooling with graph neural networks leads to practical problems. To overcome these problems, we propose two novel global graph pooling methods based on second-order pooling; namely, bilinear mapping and attentional second-order pooling. In addition, we extend attentional second-order pooling to hierarchical graph pooling for more flexible use in GNNs. We perform thorough experiments on graph classification tasks to demonstrate the effectiveness and superiority of our proposed methods. Experimental results show that our methods improve the performance significantly and consistently.

show abstract

AHNA: Adaptive representation learning for attributed heterogeneous networks

Lin

Chen

Xing

et al. 2021

Int J of Intelligent Sys

View full text Add to dashboard Cite

Meta‐path‐based random walk strategy has attracted tremendous attention in heterogeneous network representation, which can capture network semantics with heterogeneous neighborhoods of nodes. Despite the success of meta‐path‐based random walk strategy in plain heterogeneous networks which contain no attributes, it remains unexplored how meta‐path‐based random walk strategy could be utilized on attributed heterogeneous networks to simultaneously capture structural heterogeneity and attribute proximity. Moreover, the importance of node attributes and structural relations generally varies across data sets, thus requiring careful considerations when they are incorporated into representations. To tackle these problems, we propose a novel method, Attributed Heterogeneous Network embedding based on Aggregate‐path (AHNA), which generates aggregate‐path‐based random walks on attributed heterogeneous networks and adaptively fuses topological structures and node attributes based on the learned importance. Specifically, AHNA first converts node attributes to additional links in the network to deal with the heterogeneity of structures and attributes, which is followed by an adaptive random walk strategy to strike the importance balance between node attributes and topological structures, thereby generating high‐quality representations. Extensive experiments are conducted on three real‐world data sets, where AHNA outperforms state‐of‐the‐art approaches by up to 22.7%, 2.6%, and 2.3% on link prediction, community detection, and node classification, respectively. Moreover, our qualitative analysis indicates that AHNA can capture different balances of topological structures and node attributes on various data sets and thus boost the quality of node representations.

show abstract

Measuring and Relieving the Over-Smoothing Problem for Graph Neural Networks from the Topological View

Cited by 691 publications

References 11 publications

Graph Dilated Network with Rejection Mechanism

Graph Dilated Network with Rejection Mechanism

Second-Order Pooling for Graph Neural Networks

AHNA: Adaptive representation learning for attributed heterogeneous networks

Contact Info

Product

Resources

About