Deep Learning Theory Review: An Optimal Control and Dynamical Systems Perspective

Abstract

Attempts from different disciplines to provide a fundamental understanding of deep learning have advanced rapidly in recent years, yet a unified framework remains relatively limited. In this article, we provide one possible way to align existing branches of deep learning theory through the lens of dynamical system and optimal control. By viewing deep neural networks as discrete-time nonlinear dynamical systems, we can analyze how information propagates through layers using mean field theory. When optimization algorithms are further recast as controllers, the ultimate goal of training processes can be formulated as an optimal control problem. In addition, we can reveal convergence and generalization properties by studying the stochastic dynamics of optimization algorithms. This viewpoint features a wide range of theoretical study from information bottleneck to statistical physics. It also provides a principled way for hyper-parameter tuning when optimal control theory is introduced. Our framework fits nicely with supervised learning and can be extended to other learning problems, such as Bayesian learning, adversarial training, and specific forms of meta learning, without efforts. The review aims to shed lights on the importance of dynamics and optimal control when developing deep learning theory.

A Measure Theoretical Approach to the Mean-field Maximum Principle for Training NeurODEs

“…In this context, the training of the network can be interpreted as a large optimal control problem, an insight that was proposed independently by Weinan E [31] and Haber-Ruthotto [42]. Later on, this dynamical approach has been greatly popularized in the machine learning community under the name of NeurODE by Chen et al [27], see also [52]. The formulation starts by re-interpreting the iteration (1.2) as a discrete-time Euler approximation [9] of the following dynamical system Ẋt = F(t, X t , θ t ) ,…”

Generative Adversarial Network: Some Analytical Perspectives

“…Note that this FDR relation for GANs training is analogous to that for stochastic gradient descent algorithm on a pure minimization problem in [Yaida, 2019] and [Liu and Theodorou, 2019]. This FDR relation in GANs reveals the crucial difference between GANs training of discriminator and generator networks versus training of two independent neural networks.…”

Pontryagin Differentiable Programming: An End-to-End Learning and Control Framework

“…A practical training algorithm for deep learning can then be obtained by approximating the optimal solution based on the PMP via various numerical approximation (Chernousko and Lyubushin, 1982;Krylov and Chernous ḱo, 1972;LeCun, 1988). See, e.g., Benning et al (2019);E (2017);E et al (2019); Han and E (2016); Li and Hao (2018); Li et al (2017); Liu and Markowich (2019); Zhang et al (2019) for more details of these works, and see Liu and Theodorou (2019) for a recent survey on the connection between deep learning and optimal control theory.…”