Translating Numerical Concepts for PDEs into Neural Architectures

Alt, Tobias; Peter, Pascal; Weickert, Joachim; Schrader, Karl

doi:10.1007/978-3-030-75549-2_24

Cited by 8 publications

(26 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We start with simple two-dimensional diffusion models for greyscale images. Extending the connection [2,63,85] between explicit schemes for these models and residual networks [36] (ResNets) leads to neural activation functions which couple network channels. Their result is based on a rotationally invariant measure involving specific channels representing differential operators.…”

Section: Our Contributionsmentioning

confidence: 99%

“…Several works connect numerical solution strategies for PDEs to CNN architectures [2,44,46,55,86] to obtain novel architectures with better performance or provable mathematical guarantees. Others are concerned with using neural networks to solve [16,34,59] or learn PDEs from data [45,62,64].…”

Section: Related Workmentioning

confidence: 99%

“…Others are concerned with using neural networks to solve [16,34,59] or learn PDEs from data [45,62,64]. Moreover, the approximation capabilities [17,32,40,71] and stability aspects [2,10,33,61,63,70,85] of CNNs are often analysed from a PDE viewpoint.…”

Section: Related Workmentioning

confidence: 99%

“…We exploit and extend connections between variational models and diffusion processes [65], and their relations to residual networks [2,63]. In contrast to our previous works [2,4] which focussed on the one-dimensional setting and corresponding numerical algorithms, we now concentrate on two-dimensional diffusion models that incorporate different strategies to achieve rotation invariance. This allows us to transfer concepts of rotation invariance from PDEs to CNNs, which yield hitherto unexplored CNN design strategies.…”

Section: Related Workmentioning

confidence: 99%

“…In the following, we extend the connections between residual networks and explicit schemes from [2,63,85] in order to transfer rotation invariance concepts to neural networks. To this end, we consider the generalised diffusion PDE…”

Section: Coupled Activations For Operator Channelsmentioning

confidence: 99%

See 4 more Smart Citations

Designing Rotationally Invariant Neural Networks from PDEs and Variational Methods

Alt¹,

Schrader²,

Weickert³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Partial differential equation (PDE) models and their associated variational energy formulations are often rotationally invariant by design. This ensures that a rotation of the input results in a corresponding rotation of the output, which is desirable in applications such as image analysis. Convolutional neural networks (CNNs) do not share this property, and existing remedies are often complex. The goal of our paper is to investigate how diffusion and variational models achieve rotation invariance and transfer these ideas to neural networks. As a core novelty we propose activation functions which couple network channels by combining information from several oriented filters. This guarantees rotation invariance within the basic building blocks of the networks while still allowing for directional filtering. The resulting neural architectures are inherently rotationally invariant. With only a few small filters, they can achieve the same invariance as existing techniques which require a fine-grained sampling of orientations. Our findings help to translate diffusion and variational models into mathematically well-founded network architectures, and provide novel concepts for model-based CNN design.Keywords partial differential equations • variational methods • neural networks • rotation invariance • coupling

show abstract

Section: Our Contributionsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Coupled Activations For Operator Channelsmentioning

confidence: 99%

See 3 more Smart Citations

Designing Rotationally Invariant Neural Networks from PDEs and Variational Methods

Alt¹,

Schrader²,

Weickert³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

Connections Between Numerical Algorithms for PDEs and Neural Networks

et al. 2022

Self Cite

View full text Add to dashboard Cite

We investigate numerous structural connections between numerical algorithms for partial differential equations (PDEs) and neural architectures. Our goal is to transfer the rich set of mathematical foundations from the world of PDEs to neural networks. Besides structural insights, we provide concrete examples and experimental evaluations of the resulting architectures. Using the example of generalised nonlinear diffusion in 1D, we consider explicit schemes, acceleration strategies thereof, implicit schemes, and multigrid approaches. We connect these concepts to residual networks, recurrent neural networks, and U-net architectures. Our findings inspire a symmetric residual network design with provable stability guarantees and justify the effectiveness of skip connections in neural networks from a numerical perspective. Moreover, we present U-net architectures that implement multigrid techniques for learning efficient solutions of partial differential equation models, and motivate uncommon design choices such as trainable nonmonotone activation functions. Experimental evaluations show that the proposed architectures save half of the trainable parameters and can thus outperform standard ones with the same model complexity. Our considerations serve as a basis for explaining the success of popular neural architectures and provide a blueprint for developing new mathematically well-founded neural building blocks.

show abstract

PDE-Based Group Equivariant Convolutional Neural Networks

et al. 2022

View full text Add to dashboard Cite

We present a PDE-based framework that generalizes Group equivariant Convolutional Neural Networks (G-CNNs). In this framework, a network layer is seen as a set of PDE-solvers where geometrically meaningful PDE-coefficients become the layer’s trainable weights. Formulating our PDEs on homogeneous spaces allows these networks to be designed with built-in symmetries such as rotation in addition to the standard translation equivariance of CNNs. Having all the desired symmetries included in the design obviates the need to include them by means of costly techniques such as data augmentation. We will discuss our PDE-based G-CNNs (PDE-G-CNNs) in a general homogeneous space setting while also going into the specifics of our primary case of interest: roto-translation equivariance. We solve the PDE of interest by a combination of linear group convolutions and nonlinear morphological group convolutions with analytic kernel approximations that we underpin with formal theorems. Our kernel approximations allow for fast GPU-implementation of the PDE-solvers; we release our implementation with this article in the form of the LieTorch extension to PyTorch, available at https://gitlab.com/bsmetsjr/lietorch. Just like for linear convolution, a morphological convolution is specified by a kernel that we train in our PDE-G-CNNs. In PDE-G-CNNs, we do not use non-linearities such as max/min-pooling and ReLUs as they are already subsumed by morphological convolutions. We present a set of experiments to demonstrate the strength of the proposed PDE-G-CNNs in increasing the performance of deep learning-based imaging applications with far fewer parameters than traditional CNNs.

show abstract

Translating Numerical Concepts for PDEs into Neural Architectures

Cited by 8 publications

References 19 publications

Designing Rotationally Invariant Neural Networks from PDEs and Variational Methods

Designing Rotationally Invariant Neural Networks from PDEs and Variational Methods

Connections Between Numerical Algorithms for PDEs and Neural Networks

PDE-Based Group Equivariant Convolutional Neural Networks

Contact Info

Product

Resources

About