Machine Learning: Deepest Learning as Statistical Data Assimilation Problems

Abstract

We formulate an equivalence between machine learning and the formulation of statistical data assimilation as used widely in physical and biological sciences. The correspondence is that layer number in a feedforward artificial network setting is the analog of time in the data assimilation setting. This connection has been noted in the machine learning literature. We add a perspective that expands on how methods from statistical physics and aspects of Lagrangian and Hamiltonian dynamics play a role in how networks can be trained and designed. Within the discussion of this equivalence, we show that adding more layers (making the network deeper) is analogous to adding temporal resolution in a data assimilation framework. Extending this equivalence to recurrent networks is also discussed. We explore how one can find a candidate for the global minimum of the cost functions in the machine learning context using a method from data assimilation. Calculations on simple models from both sides of the equivalence are reported. Also discussed is a framework in which the time or layer label is taken to be continuous, providing a differential equation, the Euler-Lagrange equation and its boundary conditions, as a necessary condition for a minimum of the cost function. This shows that the problem being solved is a two-point boundary value problem familiar in the discussion of variational methods. The use of continuous layers is denoted "deepest learning." These problems respect a symplectic symmetry in continuous layer phase space. Both Lagrangian versions and Hamiltonian versions of these problems are presented. Their well-studied implementation in a discrete time/layer, while respecting the symplectic structure, is addressed. The Hamiltonian version provides a direct rationale for backpropagation as a solution method for a certain two-point boundary value problem.

Bayesian inference of chaotic dynamics by merging data assimilation, machine learning and expectation-maximization

“…This view of the connections and similarities between DA and ML is similar to and extents to broader configurations than the DA and ML connections put forward in [22,1,6].…”

Precision annealing Monte Carlo methods for statistical data assimilation and machine learning

“…The first sum in Eq. (7) represents the modification to π(X | Y) each time an observation y(τ k ) is made. The second sum includes the stochastic transitions of the model.…”

“…Two seemingly distinct challenges for systematically transferring information from a well curated (but noisy) data set to a model of the processes producing the data, namely statistical data assimilation (SDA) [1][2][3][4][5][6] and machine learning [7][8][9][10][11][12][13][14], have been shown to be equivalent in their formal structure [7]. In artificial neural networks, the rules that direct the activities from layer to layer are equivalent to the rules for the temporal development of dynamical models in statistical data assimilation.…”

“…We then explore the evaluation of such integrals and add to the Monte Carlo procedures a strategy to identify the dominant contribution to the expected values. We use a problem description close to SDA in physical sciences [1], nevertheless, the statements and lessons identified here are directly usable in machine learning [7].…”

Deep Learning of Markov Model Based Machines for Determination of Better Treatment Option Decisions for Infertile Women

“…As explained in our article, the machine learning and deep learning techniques broadly use the same data within the specific goals, but their approach of handling the data and models distinguish them from each other. Statistical thinking had contributed several aspects of machine learning, for example, in developing computationally intense data classification algorithms, methods in data search and matching probabilities, data mining techniques, model classification and model fitting algorithms, and a combination of all these (see for example, (17)(18)(19)(20)(21)(22)(23)(24)(25)(26)(27)(28)(29) and for a collection of articles related to statistical methods in machine learning see (30). Model-based machine learning methods (31) and the construction of coefficients in a regression model can be benefited by machine learning methods (32).…”