Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations

Abstract

Deep neural networks have become the state-of-the-art models in numerous machine learning tasks. However, general guidance to network architecture design is still missing. In our work, we bridge deep neural network design with numerical differential equations. We show that many effective networks, such as ResNet, PolyNet, FractalNet and RevNet, can be interpreted as different numerical discretizations of differential equations. This finding brings us a brand new perspective on the design of effective deep architectures. We can take advantage of the rich knowledge in numerical analysis to guide us in designing new and potentially more effective deep networks. As an example, we propose a linear multi-step architecture (LM-architecture) which is inspired by the linear multistep method solving ordinary differential equations. The LM-architecture is an effective structure that can be used on any ResNet-like networks. In particular, we demonstrate that LM-ResNet and LM-ResNeXt (i.e. the networks obtained by applying the LM-architecture on ResNet and ResNeXt respectively) can achieve noticeably higher accuracy than ResNet and ResNeXt on both CIFAR and ImageNet with comparable numbers of trainable parameters. In particular, on both CIFAR and ImageNet, LM-ResNet/LM-ResNeXt can significantly compress (> 50%) the original networks while maintaining a similar performance. This can be explained mathematically using the concept of modified equation from numerical analysis. Last but not least, we also establish a connection between stochastic control and noise injection in the training process which helps to improve generalization of the networks. Furthermore, by relating stochastic training strategy with stochastic dynamic system, we can easily apply stochastic training to the networks with the LM-architecture. As an example, we introduced stochastic depth to LM-ResNet and achieve significant improvement over the original LM-ResNet on CIFAR10.

Learning Dynamics of Attention: Human Prior for Interpretable Machine Reasoning

“…Recent work on residual networks [Lu et al, 2017, Haber and Ruthotto, 2017, Ruthotto and Haber, 2018 interpret residual connections as an Euler discretization of a continuous transformation through time. Motivated by this interpretation, Chen et al [2018] generalized residual networks by using more sophisticated black-box ODE solvers such as dopri5 [Dormand and Prince, 1980].…”

Dynamic Game Theoretic Neural Optimizer

“…Copyright 2021 by the author(s). 2017; Lu et al, 2017) to physics (Greydanus et al, 2019). For instance, casting residual networks (He et al, 2016) as a discretization of ordinary differential equations enables fundamental reasoning on the loss landscape (Lu et al, 2020) and inspires new architectures with numerical stability or continuous limit (Chang et al, 2018;Chen et al, 2018).…”

Stochastic Training of Residual Networks: a Differential Equation Viewpoint

“…Introduction. Artificial neural networks are appearing as the state-of-the-art technologies in many machine learning tasks including, but not limited to, computer vision, speech recognition, and natural language processing [20,29], among which the residual network (ResNet) and its numerous variants (see [9,7,14,25,36] and references cited therein) have attracted broad attention for their simplicity and effectiveness. In addition, ResNet makes it possible to train up to hundreds or even thousands of layers without performance degradation [10].…”

“…However, deeper networks usually require extensive training, thereby impeding their real-time applications. To circumvent this issue as well as improve the generalization capability of trained models, the use of stochastic training techniques has become widespread in deep learning community [15,25]. Unfortunately, design of the data-oriented network architecture for real-world learning problems is often more art than science, e.g., injection of the dropout layer into ResNet-like models [10,36], tuning of the dropout probability and model hyperparameters [15,32], etc.…”

“…Unfortunately, design of the data-oriented network architecture for real-world learning problems is often more art than science, e.g., injection of the dropout layer into ResNet-like models [10,36], tuning of the dropout probability and model hyperparameters [15,32], etc. As such, the latest studies have concentrated on the continuous-time dynamic of ResNets via (stochastic) modified equations [24,25,35], which is highly desirable in order to explore the nature of deep ResNets and construct potentially more effective ones.…”