Connections between Numerical Algorithms for PDEs and Neural Networks

Abstract

We investigate numerous structural connections between numerical algorithms for partial differential equations (PDEs) and neural architectures. Our goal is to transfer the rich set of mathematical foundations from the world of PDEs to neural networks. Besides structural insights we provide concrete examples and experimental evaluations of the resulting architectures. Using the example of generalised nonlinear diffusion in 1D, we consider explicit schemes, acceleration strategies thereof, implicit schemes, and multigrid approaches. We connect these concepts to residual networks, recurrent neural networks, and U-net architectures. Our findings inspire a symmetric residual network design with provable stability guarantees and justify the effectiveness of skip connections in neural networks from a numerical perspective. Moreover, we present U-net architectures that implement multigrid techniques for learning efficient solutions of partial differential equation models, and motivate uncommon design choices such as trainable nonmonotone activation functions. Experimental evaluations show that the proposed architectures save half of the trainable parameters and can thus outperform standard ones with the same model complexity. Our considerations serve as a basis for explaining the success of popular neural architectures and provide a blueprint for developing new mathematically well-founded neural building blocks.

Efficient Long-Term Simulation of the Heat Equation with Application in Geothermal Energy Storage

“…Although FSI requires an additional computation of the sum of two vectors, the computational costs are less than FED because non-varying time step sizes are used within the matrix-vector multiplications in (34), meaning I + τL must be computed only once. Another benefit of FSI is its straightforward use for nonlinear problems L(u k ), where the scheme allows us to perform nonlinear updates within one cycle.…”

“…However, the schemes can be applied to many parabolic problems, including time-dependent boundary condition, also in an engineering context as demonstrated in this work. Beyond this, FSI can also be translated to neural architectures and thus offers a high level of practical relevance, we refer the interested reader to the current work [34].…”

Learning phase field mean curvature flows with neural networks

“…Another approach considers a neural network as an operator between Euclidean spaces of same dimension depending on the discretization of the PDE [29,30,31,32,33]. This approach depends on the discretization and requires to modify the architecture of the network when the discrete resolution or the discretization are changed; • Most neural network architectures can be interpreted as numerical schemes [34,35]. Neural networks can therefore be seen as operators acting between infinite-dimensional spaces (typically spaces of functions): for instance, for a time-dependent PDE, the forward propagation of an associated neural network can be viewed as the flow associated to the PDE when a time-step δ t is fixed.…”

Quantized convolutional neural networks through the lens of partial differential equations

“…We show that by integrating TV smoothing steps into existing network architectures (as a non-pointwise activation), we improve the performance of CNNs in classification and semantic segmentation tasks. A related TV approach was suggested very recently in [2], albeit not in the context of quantization, and using a learnt activation function. Next, we examine the behaviour of quantization under symmetric and stable, heat equation-like CNNs [32,1,2].…”

“…A related TV approach was suggested very recently in [2], albeit not in the context of quantization, and using a learnt activation function. Next, we examine the behaviour of quantization under symmetric and stable, heat equation-like CNNs [32,1,2]. We show that the quantization process produces significantly lighter-weight networks, in terms of storage and computation, while only incurring a minimal loss of accuracy.…”