Convergent Boosted Smoothing for Modeling Graph Data with Tabular Node Features

Abstract

For supervised learning with tabular data, decision tree ensembles produced via boosting techniques generally dominate real-world applications involving iid training/test sets. However for graph data where the iid assumption is violated due to structured relations between samples, it remains unclear how to best incorporate this structure within existing boosting pipelines. To this end, we propose a generalized framework for iterating boosting with graph propagation steps that share node/sample information across edges connecting related samples. Unlike previous efforts to integrate graph-based models with boosting, our approach is anchored in a principled meta loss function such that provable convergence can be guaranteed under relatively mild assumptions. Across a variety of non-iid graph datasets with tabular node features, our method achieves comparable or superior performance than both tabular and graph neural network models, as well as existing hybrid strategies that combine the two. Beyond producing better predictive performance than recently proposed graph models, our proposed techniques are easy to implement, computationally more efficient, and enjoy stronger theoretical guarantees (which make our results more reproducible).

A Robust Stacking Framework for Training Deep Graph Models with Multifaceted Node Features

“…Leading GNN models fail to achieve competitive results for heterogeneous features with tabular or text node features [Ivanov and Prokhorenkova, 2021, Huang et al, 2020, Chen et al, 2021. To remedy this, Ivanov and Prokhorenkova [2021] jointly train Gradient Boosted Decision Trees (GBDT) and GNN in an end-to-end fashion, demonstrating a significant increase in performance on graph data with tabular node features.…”

“…To remedy this, Ivanov and Prokhorenkova [2021] jointly train Gradient Boosted Decision Trees (GBDT) and GNN in an end-to-end fashion, demonstrating a significant increase in performance on graph data with tabular node features. Chen et al [2021] removes the need for a GNN altogether, proposing a generalized framework for iterating boosting with parameter-free graph propagation steps that share node/sample information across edges connecting related samples.…”

“…Correct and Smooth (C&S) [Huang et al, 2020] is a simple post-processing step that applies label propagation to further incorporate graph information into the outputs of a learning algorithm. Chen et al [2021] trains Gradient Boosted Decision Trees with label propagation incorporated into the objective function, producing competitive results for graph data with tabular node features.…”

“…However GNNs have various limitations, in particular they require appropriately setting many hyperparameters and are nontrivial to apply to graph data in which the node features are not completely numerical. For graph data with tabular (numeric and categorical) features, it has been observed that certain models for IID data (which ignore the graph structure) can be competitive with GNNs, if these models are combined with simple propagation operations to account for the graph structure [Huang et al, 2020, Chen et al, 2021. We use the shorthand IID models to refer to these models intended for IID data, which in our setting simply operate on one node's features as if they were independent of the other nodes' features (of course this independence does not actually hold for graph data).…”