Learning to Pool in Graph Neural Networks for Extrapolation

Abstract

Graph neural networks (GNNs) are one of the most popular approaches to using deep learning on graph-structured data, and they have shown state-of-the-art performances on a variety of tasks. However, according to a recent study, a careful choice of pooling functions, which are used for the aggregation or readout operation in GNNs, is crucial for enabling GNNs to extrapolate. Without the ideal combination of pooling functions, which varies across tasks, GNNs completely fail to generalize to out-of-distribution data, while the number of possible combinations grows exponentially with the number of layers. In this paper, we present GNP, a L p norm-like pooling function that is trainable end-to-end for any given task. Notably, GNP generalizes most of the widely-used pooling functions. We verify experimentally that simply replacing all pooling functions with GNP enables GNNs to extrapolate well on many node-level, graph-level, and set-related tasks; and GNP sometimes performs even better than optimal combinations of existing pooling functions.

Regularized Optimal Transport Layers for Generalized Global Pooling Operations

“…For ROTP B-E and ROTP B-Q , we make the α 0 learnable. The baselines we considered include i) classic pooling operations like Add-Pooling, Mean-Pooling, and Max-Pooling; ii) the mixed pooling operations like the Mixed Mean-Max and the Gated Mean-Max in [10]; iii) the learnable global pooling layers like DeepSet [15], Set2Set [14], DynamicPooling [1], GNP [28], and the Attention-Pooling and Gated Attention in [2]; iv) the attention-pooling methods for graphs, i.e., SAGPooling [33], AS-APooling [32]; and v) OT-based pooling methods, i.e., OTK [59], WEGL [63], and SWE [62]. The above pooling methods are trained and tested on a server with two Nvidia RTX3090 GPUs, whose key hyperparameters are set by grid search.…”

“…In [10], the mixed mean-max pooling and its structured variants leverage mixture models of mean-pooling and max-pooling to improve pooling results. Recently, a generalized norm-based pooling (GNP) is proposed in [28], [29]. It can imitate max-pooling and mean-pooling under different settings.…”

“…In practice, X can be used to represent the instances in a bag, the node embeddings of a graph, or the local visual features in an image. Following the work in [28], [29], [73], we assume the input data X to be nonnegative in this study. This assumption is generally reasonable because the input data are often processed by non-negative activations, like ReLU, Sigmoid, Softplus, and so on.…”

“…This assumption is generally reasonable because the input data are often processed by non-negative activations, like ReLU, Sigmoid, Softplus, and so on. For some pooling methods, e.g., the max-pooling and the generalized norm pooling [28], the nonnegativeness of input data is even necessary.…”