Stochastic Training of Residual Networks: a Differential Equation Viewpoint

Sun, Qizhen; Tao, Yunzhe; Du, Qiang

doi:10.48550/arxiv.1812.00174

Cited by 9 publications

(10 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Clearly, the framework of the NODEs can be formulated as a typical problem of optimal control on ODEs. Additionally, the framework of NODEs has been generalized to the other dynamical systems, such as the Partial Differential Equations (PDEs) (Han et al, 2018;Long et al, 2018;Ruthotto & Haber, 2019;Sun et al, 2020) and the Stochastic Differential Equations (SDEs) Sun et al, 2018;, where the theory of optimal control has been completely established. It is worthwhile to mention that the optimal control theory is tightly connected with and benefits from the method of the classical calculus of variations (Liberzon, 2011).…”

Section: Related Workmentioning

confidence: 99%

Neural Delay Differential Equations

Zhu¹,

Guo²,

Lin³

2021

Preprint

View full text Add to dashboard Cite

Neural Ordinary Differential Equations (NODEs), a framework of continuousdepth neural networks, have been widely applied, showing exceptional efficacy in coping with some representative datasets. Recently, an augmented framework has been successfully developed for conquering some limitations emergent in application of the original framework. Here we propose a new class of continuous-depth neural networks with delay, named as Neural Delay Differential Equations (ND-DEs), and, for computing the corresponding gradients, we use the adjoint sensitivity method to obtain the delayed dynamics of the adjoint. Since the differential equations with delays are usually seen as dynamical systems of infinite dimension possessing more fruitful dynamics, the NDDEs, compared to the NODEs, own a stronger capacity of nonlinear representations. Indeed, we analytically validate that the NDDEs are of universal approximators, and further articulate an extension of the NDDEs, where the initial function of the NDDEs is supposed to satisfy ODEs. More importantly, we use several illustrative examples to demonstrate the outstanding capacities of the NDDEs and the NDDEs with ODEs' initial value. Specifically, (1) we successfully model the delayed dynamics where the trajectories in the lower-dimensional phase space could be mutually intersected, while the traditional NODEs without any argumentation are not directly applicable for such modeling, and (2) we achieve lower loss and higher accuracy not only for the data produced synthetically by complex models but also for the real-world image datasets, i.e., CIFAR10, MNIST, and SVHN. Our results on the NDDEs reveal that appropriately articulating the elements of dynamical systems into the network design is truly beneficial to promoting the network performance.

show abstract

Section: Related Workmentioning

confidence: 99%

Neural Delay Differential Equations

Zhu¹,

Guo²,

Lin³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…can indeed be seen as a discretized Euler scheme with unit step size of the ordinary differential equation (ODE) ẋi = T (x i ) (Weinan, 2017;Chen et al, 2018;Teh et al, 2019;Sun et al, 2018;Weinan et al, 2019;Lu et al, 2018;Ruthotto and Haber, 2019;Sander et al, 2021). In section 4, we adopt this point of view on residual attention layers in order to get a better theoretical understanding of attention mechanisms.…”

Section: Background and Related Workmentioning

confidence: 99%

Sinkformers: Transformers with Doubly Stochastic Attention

E.¹,

Ablin²,

Blondel³

et al. 2021

Preprint

View full text Add to dashboard Cite

Attention based models such as Transformers involve pairwise interactions between data points, modeled with a learnable attention matrix. Importantly, this attention matrix is normalized with the SoftMax operator, which makes it row-wise stochastic. In this paper, we propose instead to use Sinkhorn's algorithm to make attention matrices doubly stochastic. We call the resulting model a Sinkformer. We show that the row-wise stochastic attention matrices in classical Transformers get close to doubly stochastic matrices as the number of epochs increases, justifying the use of Sinkhorn normalization as an informative prior. On the theoretical side, we show that, unlike the SoftMax operation, this normalization makes it possible to understand the iterations of self-attention modules as a discretized gradient-flow for the Wasserstein metric. We also show in the infinite number of samples limit that, when rescaling both attention matrices and depth, Sinkformers operate a heat diffusion. On the experimental side, we show that Sinkformers enhance model accuracy in vision and natural language processing tasks. In particular, on 3D shapes classification, Sinkformers lead to a significant improvement.

show abstract

“…Accordingly, Novak et al [13] distinguished different network models based on their sensitivity and came up with a robustness measure through their input-output Jacobian. Alternatively, several researchers have connected neural networks to dynamical models to estimate the complexity from a theoretical perspective by establishing a parallel between the network architectures and stochastic partial differential equations [20], [21], [22]. A different approach, like Zhang et al [23], is focused on complexity estimation of recurrent neural networks by using the concept of cyclic graph to define recurrent depth or recurrent skip coefficient to capture how rapidly information propagates over time.…”

Section: Related Workmentioning

confidence: 99%

Neural Network Layer Algebra: A Framework to Measure Capacity and Compression in Deep Learning

Badías¹,

Banerjee²

2021

Preprint

View full text Add to dashboard Cite

We present a new framework to measure the intrinsic properties of (deep) neural networks. While we focus on convolutional networks, our framework can be extrapolated to any network architecture. In particular, we evaluate two network properties, namely, capacity (related to expressivity) and compression, both of which depend only on the network structure and are independent of the training and test data. To this end, we propose two metrics: the first one, called layer complexity, captures the architectural complexity of any network layer; and, the second one, called layer intrinsic power, encodes how data is compressed along the network. The metrics are based on the concept of layer algebra, which is also introduced in this paper. This concept is based on the idea that the global properties depend on the network topology, and the leaf nodes of any neural network can be approximated using local transfer functions, thereby, allowing a simple computation of the global metrics. We also compare the properties of the state-of-the art architectures using our metrics and use the properties to analyze the classification accuracy on benchmark datasets.

show abstract

Stochastic Training of Residual Networks: a Differential Equation Viewpoint

Cited by 9 publications

References 21 publications

Neural Delay Differential Equations

Neural Delay Differential Equations

Sinkformers: Transformers with Doubly Stochastic Attention

Neural Network Layer Algebra: A Framework to Measure Capacity and Compression in Deep Learning

Contact Info

Product

Resources

About