Analytically Embedding Differential Equation Constraints into Least Squares Support Vector Machines Using the Theory of Functional Connections

Abstract

Differential equations (DEs) are used as numerical models to describe physical phenomena throughout the field of engineering and science, including heat and fluid flow, structural bending, and systems dynamics. While there are many other techniques for finding approximate solutions to these equations, this paper looks to compare the application of the Theory of Functional Connections (TFC) with one based on least-squares support vector machines (LS-SVM). The TFC method uses a constrained expression, an expression that always satisfies the DE constraints, which transforms the process of solving a DE into solving an unconstrained optimization problem that is ultimately solved via least-squares (LS). In addition to individual analysis, the two methods are merged into a new methodology, called constrained SVMs (CSVM), by incorporating the LS-SVM method into the TFC framework to solve unconstrained problems. Numerical tests are conducted on four sample problems: One first order linear ordinary differential equation (ODE), one first order nonlinear ODE, one second order linear ODE, and one two-dimensional linear partial differential equation (PDE). Using the LS-SVM method as a benchmark, a speed comparison is made for all the problems by timing the training period, and an accuracy comparison is made using the maximum error and mean squared error on the training and test sets. In general, TFC is shown to be slightly faster (by an order of magnitude or less) and more accurate (by multiple orders of magnitude) than the LS-SVM and CSVM approaches. IntroductionDifferential equations (DE) and their solutions are important topics in science and engineering. The solutions drive the design of predictive systems models and optimization tools. Currently, these equations are solved by a variety of existing approaches with the most popular based on the Runge-Kutta family [1]. Other methods include those which leverage low-order Taylor expansions, namely Gauss-Jackson [2] and Chebyshev-Picard iteration [3,4,5], which have proven to be highly effective. More recently developed techniques are based on spectral collocation methods [6]. This approach discretizes the domain about collocation points, and the solution of the DE is expressed by a sum of "basis" functions with unknown coefficients that are approximated in order to satisfy the DE as closely as possible. Yet, in order to incorporate boundary conditions, one or more equations must be added to enforce the constraints.The Theory of Functional Connections (TFC) is a new technique that analytically derives a constrained expression which satisfies the problem's constraints exactly while maintaining a function that can be freely chosen [7]. This theory, initially called "Theory of Connections", has been renamed for two reasons. First, the "Theory of Connections" already identifies a specific theory in differential geometry, and second, what this theory is actually doing is "functional interpolation", as it provides all functions satisfying a set of constraints in terms of a function ...

Deep Theory of Functional Connections: A New Method for Estimating the Solutions of Partial Differential Equations

“…For the reader's reference, the assumed solution form for problem 1 from Ref. [5] is shown in Equation (10). Equation (10) was copied from Ref.…”

“…In Ref. [10], TFC was used to embed constraints into support vector machines, but left embedding constraints into neural networks to future work. This research shows how to embed constraints into neural networks with the TFC, and leverages this technique to numerically estimate the solutions of PDEs.…”

The Multivariate Theory of Functional Connections: Theory, Proofs, and Application in Partial Differential Equations

“…However, Reference [9] does not discuss how the multivariate framework can be used to construct constrained expressions for linear constraints. Regardless, these multivariate constrained expressions have been used to embed constraints into machine learning frameworks [10][11][12] for use in solving partial differential equations (PDEs). Moreover, it was shown that this framework may be combined with orthogonal basis functions to solve PDEs [13]; this is essentially the n-dimensional equivalent of the ordinary differential equations (ODEs) solved using the univariate formulation [2,3].…”

Bijective Mapping Analysis to Extend the Theory of Functional Connections to Non-Rectangular 2-Dimensional Domains

“…A mathematical proof [1] has shown that constrained expressions, like the one given in Equation (1), can be used to represent the whole set of functions satisfying the set of constraints they are derived for. The TFC has been mainly developed [1][2][3][4] to better solve constraint optimization problems, such as ODEs [5][6][7][8], PDEs [4,9], or programming [10,11], with effective applications in optimal control [12,13], as well as in machine learning [4,14,15]. In fact, a constrained expression restricts the whole space of functions to the subspace fully satisfying the constraints.…”

“…The TFC has been mainly developed [ 1 – 4 ] to better solve constraint optimization problems, such as ODEs [ 5 – 8 ], PDEs [ 4 , 9 ], or programming [ 10 , 11 ], with effective applications in optimal control [ 12 , 13 ], as well as in machine learning [ 4 , 14 , 15 ]. In fact, a constrained expression restricts the whole space of functions to the subspace fully satisfying the constraints.…”