Lifting Layers: Analysis and Applications

Ochs, Peter; Meinhardt, Tim; Leal-Taixé, Laura; Moeller, Michael

doi:10.1007/978-3-030-01246-5_4

Cited by 9 publications

(7 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Nonmonotone activation functions are rarely found in standard CNNs, with some notable exceptions [20,38,76]. Recently, the so-called Swish activation [84] and modifications thereof [72,112] have been found to empirically boost the classification performance of CNNs.…”

Section: Related Workmentioning

confidence: 99%

Connections Between Numerical Algorithms for PDEs and Neural Networks

et al. 2022

View full text Add to dashboard Cite

We investigate numerous structural connections between numerical algorithms for partial differential equations (PDEs) and neural architectures. Our goal is to transfer the rich set of mathematical foundations from the world of PDEs to neural networks. Besides structural insights, we provide concrete examples and experimental evaluations of the resulting architectures. Using the example of generalised nonlinear diffusion in 1D, we consider explicit schemes, acceleration strategies thereof, implicit schemes, and multigrid approaches. We connect these concepts to residual networks, recurrent neural networks, and U-net architectures. Our findings inspire a symmetric residual network design with provable stability guarantees and justify the effectiveness of skip connections in neural networks from a numerical perspective. Moreover, we present U-net architectures that implement multigrid techniques for learning efficient solutions of partial differential equation models, and motivate uncommon design choices such as trainable nonmonotone activation functions. Experimental evaluations show that the proposed architectures save half of the trainable parameters and can thus outperform standard ones with the same model complexity. Our considerations serve as a basis for explaining the success of popular neural architectures and provide a blueprint for developing new mathematically well-founded neural building blocks.

show abstract

Section: Related Workmentioning

confidence: 99%

Connections Between Numerical Algorithms for PDEs and Neural Networks

et al. 2022

View full text Add to dashboard Cite

show abstract

“…The diffusion interpretation suggests that activation functions should be learned in the same manner as convolution weights and biases. In practice, this hardly happens apart from a few notable exceptions such as [3,7,17]. As nonmonotone flux functions outperform monotone ones in the diffusion setting, it appears promising to incorporate them into CNNs.…”

Section: Nonmonotone Activation Functionsmentioning

confidence: 99%

Translating Numerical Concepts for PDEs into Neural Architectures

Alt¹,

Peter²,

Weickert³

et al. 2021

Preprint

View full text Add to dashboard Cite

We investigate what can be learned from translating numerical algorithms into neural networks. On the numerical side, we consider explicit, accelerated explicit, and implicit schemes for a general higher order nonlinear diffusion equation in 1D, as well as linear multigrid methods. On the neural network side, we identify corresponding concepts in terms of residual networks (ResNets), recurrent networks, and U-nets. These connections guarantee Euclidean stability of specific ResNets with a transposed convolution layer structure in each block. We present three numerical justifications for skip connections: as time discretisations in explicit schemes, as extrapolation mechanisms for accelerating those methods, and as recurrent connections in fixed point solvers for implicit schemes. Last but not least, we also motivate uncommon design choices such as nonmonotone activation functions. Our findings give a numerical perspective on the success of modern neural network architectures, and they provide design criteria for stable networks.

show abstract

“…Nonmonotone activation functions are rarely found in standard CNNs, with some notable exceptions [18,35,72]. Recently, the so-called Swish activation [80] and modifications thereof [68,108] have been found to empirically boost the classification performance of CNNs.…”

Section: Related Workmentioning

confidence: 99%

“…In practice, this hardly happens apart from a few notable exceptions such as [18,35,72]. Recently, activation functions which are slightly nonmonotone variants of the ReLU proved successful for image classification tasks [68,80,108].…”

Section: Relu Activation Charbonnier Activationmentioning

confidence: 99%

Connections between Numerical Algorithms for PDEs and Neural Networks

Alt¹,

Schrader²,

Augustin³

et al. 2021

Preprint

View full text Add to dashboard Cite

We investigate numerous structural connections between numerical algorithms for partial differential equations (PDEs) and neural architectures. Our goal is to transfer the rich set of mathematical foundations from the world of PDEs to neural networks. Besides structural insights we provide concrete examples and experimental evaluations of the resulting architectures. Using the example of generalised nonlinear diffusion in 1D, we consider explicit schemes, acceleration strategies thereof, implicit schemes, and multigrid approaches. We connect these concepts to residual networks, recurrent neural networks, and U-net architectures. Our findings inspire a symmetric residual network design with provable stability guarantees and justify the effectiveness of skip connections in neural networks from a numerical perspective. Moreover, we present U-net architectures that implement multigrid techniques for learning efficient solutions of partial differential equation models, and motivate uncommon design choices such as trainable nonmonotone activation functions. Experimental evaluations show that the proposed architectures save half of the trainable parameters and can thus outperform standard ones with the same model complexity. Our considerations serve as a basis for explaining the success of popular neural architectures and provide a blueprint for developing new mathematically well-founded neural building blocks.

show abstract

Lifting Layers: Analysis and Applications

Cited by 9 publications

References 16 publications

Connections Between Numerical Algorithms for PDEs and Neural Networks

Connections Between Numerical Algorithms for PDEs and Neural Networks

Translating Numerical Concepts for PDEs into Neural Architectures

Connections between Numerical Algorithms for PDEs and Neural Networks

Contact Info

Product

Resources

About