On Generalization Bounds of a Family of Recurrent Neural Networks

Chen, Minshuo; Li, Xingguo; Zhao, Tuo

doi:10.48550/arxiv.1910.12947

Cited by 8 publications

(12 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Next, we study the generalization ability of GNNs via Rademacher bounds, focusing on binary classification. We generalize the previous results on the complexity of feedforward networks (Bartlett et al, 2017;Neyshabur et al, 2018) and RNNs (Chen et al, 2019a) in mainly three ways. First, we process graphs unlike sequences in RNNs, or instances restricted to the input layer in feedforward networks.…”

Section: Generalization Bounds For Gnnssupporting

confidence: 79%

“…We also mention the corresponding bounds for RNN on a sequence of length L when the spectral norm of recurrent weights in RNN is respectively less than, equal to, or greater than 1 (note that we renamed some parameters from Chen et al (2019a) for notational consistency). Our analysis implies GNNs have essentially the same dependence on dimension r, C GNN (ours) RNN (Chen et al (2019a))…”

Section: Generalization Bound For Gnnsmentioning

confidence: 69%

“…Generalization. Several works have established generalization guarantees for FFNs (Bartlett et al, 2017;Golowich et al, 2018;Neyshabur et al, 2018;Zhang et al, 2018a) and RNNs (Chen et al, 2019a;Allen-Zhu and Li, 2019). GNNs differ in some key aspects from those models.…”

Section: Related Workmentioning

confidence: 99%

“…Using Lemma 8 from (Chen et al, 2019a), we obtain the corresponding bounds on their covering number. Specifically,…”

Section: Proof Of Lemmamentioning

confidence: 99%

See 3 more Smart Citations

Generalization and Representational Limits of Graph Neural Networks

Garg,

Jegelka,

Jaakkola

2020

Preprint

View full text Add to dashboard Cite

We address two fundamental questions about graph neural networks (GNNs). First, we prove that several important graph properties cannot be computed by GNNs that rely entirely on local information. Such GNNs include the standard message passing models, and more powerful spatial variants that exploit local graph structure (e.g., via relative orientation of messages, or local port ordering) to distinguish neighbors of each node. Our treatment includes a novel graph-theoretic formalism. Second, we provide the first data dependent generalization bounds for message passing GNNs. This analysis explicitly accounts for the local permutation invariance of GNNs. Our bounds are much tighter than existing VC-dimension based guarantees for GNNs, and are comparable to Rademacher bounds for recurrent neural networks.

show abstract

Section: Generalization Bounds For Gnnssupporting

confidence: 79%

Section: Generalization Bound For Gnnsmentioning

confidence: 69%

Section: Related Workmentioning

confidence: 99%

“…Using Lemma 8 from (Chen et al, 2019a), we obtain the corresponding bounds on their covering number. Specifically,…”

Section: Proof Of Lemmamentioning

confidence: 99%

See 2 more Smart Citations

Generalization and Representational Limits of Graph Neural Networks

Garg,

Jegelka,

Jaakkola

2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Most recently, Sato et al (2020) provides PAC learning-style bounds on the node embedding and gradient estimation for SGCNs training. Another direction of theoretical research focuses on analyzing the expressive power of GCN (Garg et al, 2020;Chen et al, 2019;, which is not the focus of this paper and omitted for brevity.…”

mentioning

confidence: 99%

On the Importance of Sampling in Training GCNs: Tighter Analysis and Variance Reduction

Cong

Ramezani

Mahdavi

2021

Preprint

View full text Add to dashboard Cite

Graph Convolutional Networks (GCNs) have achieved impressive empirical advancement across a wide variety of graph-related applications. Despite their great success, training GCNs on large graphs su ers from computational and memory issues. A potential path to circumvent these obstacles is sampling-based methods, where at each layer a subset of nodes is sampled. Although recent studies have empirically demonstrated the e ectiveness of sampling-based methods, these works lack theoretical convergence guarantees under realistic settings and cannot fully leverage the information of evolving parameters during optimization. In this paper, we describe and analyze a general doubly variance reduction schema that can accelerate any sampling method under the memory budget. The motivating impetus for the proposed schema is a careful analysis for the variance of sampling methods where it is shown that the induced variance can be decomposed into node embedding approximation variance (zeroth-order variance) during forward propagation and layerwise-gradient variance ( rst-order variance) during backward propagation. We theoretically analyze the convergence of the proposed schema and show that it enjoys an (1/ ) convergence rate. We complement our theoretical results by integrating the proposed schema in di erent sampling methods and applying them to di erent large real-world graphs. Code is public available at https://github.com/CongWeilin/SGCN.git.

show abstract

Statistical machine‐learning–based predictive control of uncertain nonlinear processes

Alnajdi²,

et al. 2022

AIChE Journal

View full text Add to dashboard Cite

In this study, we present machine-learning-based predictive control schemes for nonlinear processes subject to disturbances, and establish closed-loop system stability properties using statistical machine learning theory. Specifically, we derive a generalization error bound via Rademacher complexity method for the recurrent neural networks (RNN) that are developed to capture the dynamics of the nominal system.Then, the RNN models are incorporated in Lyapunov-based model predictive controllers, under which we study closed-loop stability properties for the nonlinear systems subject to two types of disturbances: bounded disturbances and stochastic disturbances with unbounded variation. A chemical reactor example is used to demonstrate the implementation and evaluate the performance of the proposed approach.

show abstract

On Generalization Bounds of a Family of Recurrent Neural Networks

Cited by 8 publications

References 29 publications

Generalization and Representational Limits of Graph Neural Networks

Generalization and Representational Limits of Graph Neural Networks

On the Importance of Sampling in Training GCNs: Tighter Analysis and Variance Reduction

Statistical machine‐learning–based predictive control of uncertain nonlinear processes

Contact Info

Product

Resources

About