New optimization algorithms for neural network training using operator splitting techniques

The long time behavior and the rate of convergence of symplectic convex algorithms obtained via splitting discretizations of inertial damping systems

“…For the (HBF γ ) the objective function f does not decrease along the trajectories. On the other hand, the global energy (kinetic + potential) E (t) = 1 2 ẋ(t) 2 + f (x(t)) is monotone decreasing, by being a continuous Lyapunov-type functional. Furthermore, we remind that Attouch.…”

“…If the vector field from the right hand side can be decomposed into two vector fields, i.e. F = F [1] + F [2] , then we attach the associated sub-problems of (5), namely [1] (ϕ(t)) and (P 2 ) : d dt ϕ(t) = F [2] (ϕ(t))…”

“…Let ϕ [1] = ϕ [1] (t) be the solution of (P 1 ) and ϕ [2] = ϕ [2] (t) be the solution of (P 2 ). Then, the Lie-Trotter splitting or sequential splitting (see [41]) after one stepsize h is :…”

“…Applying recursively, we obtain the splitting operator formula for the discrete time t n = nh. Quite interestingly, if, for example, the exact flows ϕ [1] (t) and ϕ [2] (t) are not available, then one can use consistent approximations to these solutions. In this manner, from now on, φ [1] h and φ [2] h represents numerical discretization of the flows on the two sub-problems (P 1 ) and (P 2 ) at time t n = nh, respectively.…”

“…Quite interestingly, if, for example, the exact flows ϕ [1] (t) and ϕ [2] (t) are not available, then one can use consistent approximations to these solutions. In this manner, from now on, φ [1] h and φ [2] h represents numerical discretization of the flows on the two sub-problems (P 1 ) and (P 2 ) at time t n = nh, respectively. Then, for example the Lie-Trotter splitting (LTS 1 ) can be defined as…”