Fractional deep neural network via constrained optimization

Abstract

This paper introduces a novel algorithmic framework for a deep neural network (DNN), which in a mathematically rigorous manner, allows us to incorporate history (or memory) into the network—it ensures all layers are connected to one another. This DNN, called Fractional-DNN, can be viewed as a time-discretization of a fractional in time non-linear ordinary differential equation (ODE). The learning problem then is a minimization problem subject to that fractional ODE as constraints. We emphasize that an analogy between the existing DNN and ODEs, with standard time derivative, is well-known by now. The focus of our work is the Fractional-DNN. Using the Lagrangian approach, we provide a derivation of the backward propagation and the design equations. We test our network on several datasets for classification problems. Fractional-DNN offers various advantages over the existing DNN. The key benefits are a significant improvement to the vanishing gradient issue due to the memory effect, and better handling of nonsmooth data due to the network’s ability to approximate non-smooth functions.

Artificial neural networks: a practical review of applications involving fractional calculus

“…Besides, Antil in Ref. [50] proposed a deep FANN for the time-discretization of a fractional in time nonlinear ordinary DE, employing Caputo derivative. The fractional ordinary DE was minimized by the learning algorithm, solving several issues, such as network instability, vanishing, exploding gradients, long training times, and inability to approximate non-smooth functions.…”

Deep learning or interpolation for inverse modelling of heat and fluid flow problems?

“…Deep learning (DL), a new Artificial Intelligence (AI) trend that uses multi-layer perception network [8], has received increasing attention from researchers and has been widely applied to numerous real-world applications and across many fields [2,9,10,11,12,13,14,15]. Deep learning is able to effectively capture the non-linear and non-trivial user-item relationships, and it enables the codification of more complex abstractions as data representations in the higher layers.…”

Novel DNNs for Stiff ODEs with Applications to Chemically Reacting Flows

“…The DNNs used in this paper have been motivated by the Residual Neural Network (ResNet) architecture. ResNets have been introduced in [9, 24,2] in the context of data/image classification, see also [1] for parameterized PDEs and [7] where the (related) so-called Neural ODE Nets [4] have been used to solve stiff ODEs. The ResNet architecture is known to overcome the vanishing gradient problem, which has been further analyzed using fractional order derivatives in [2].…”

“…ResNets have been introduced in [9, 24,2] in the context of data/image classification, see also [1] for parameterized PDEs and [7] where the (related) so-called Neural ODE Nets [4] have been used to solve stiff ODEs. The ResNet architecture is known to overcome the vanishing gradient problem, which has been further analyzed using fractional order derivatives in [2]. The key feature of a ResNet is that in the continuum limit, it becomes an optimization problem constrained by an ODE.…”

“…• These networks allow for learning both the solution (see (1)) and difference quotients (see (2)). A similar approach to learn the solution in the context of parameterized PDEs has been recently considered in [1].…”