Approximation Properties of Deep ReLU CNNs

He, Juncai; Li, Lin; Xu, Jinchao

doi:10.48550/arxiv.2109.00190

Cited by 4 publications

(5 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…An alternative is the convolutional autoencoder (CAE). Convolutional networks not only have the characteristics, such as the connectivity, the translation invariance and so on [17,28], but also has well approximation properties [3,20,38,48]. Refs.…”

Section: Convolutional Autoencoder Based Nonlinear Rommentioning

confidence: 99%

Solving Time-Dependent Parametric PDEs by Multiclass Classification-Based Reduced Order Model

null¹,

Shu²

2023

CSIAM-AM

View full text Add to dashboard Cite

In this paper, we propose a network model, the multiclass classificationbased reduced order model (MC-ROM), for solving time-dependent parametric partial differential equations (PPDEs). This work is inspired by the observation of applying the deep learning-based reduced order model (DL-ROM) [14] to solve diffusiondominant PPDEs. We find that the DL-ROM has a good approximation for some special model parameters, but it cannot approximate the drastic changes of the solution as time evolves. Based on this fact, we classify the dataset according to the magnitude of the solutions and construct corresponding subnets dependent on different types of data. Then we train a classifier to integrate different subnets together to obtain the MC-ROM. When subsets have the same architecture, we can use transfer learning techniques to accelerate offline training. Numerical experiments show that the MC-ROM improves the generalization ability of the DL-ROM both for diffusion-and convectiondominant problems, and maintains the DL-ROM's advantage of good approximation ability. We also compare the approximation accuracy and computational efficiency of the proper orthogonal decomposition (POD) which is not suitable for convectiondominant problems. For diffusion-dominant problems, the MC-ROM has better approximation accuracy than the POD in a small dimensionality reduction space, and its computational performance is more efficient than the POD's.

show abstract

Section: Convolutional Autoencoder Based Nonlinear Rommentioning

confidence: 99%

Solving Time-Dependent Parametric PDEs by Multiclass Classification-Based Reduced Order Model

null¹,

Shu²

2023

CSIAM-AM

View full text Add to dashboard Cite

show abstract

“…Gühring, Kutyniok, and Petersen (2020); Gühring and Raslan (2021); Siegel and Xu (2022), and adapted to the case of CNNs exploiting some algebraic arguments that link dense and convolutional layers, see e.g. Zhou (2020); He, Li, and Xu (2021).…”

Section: Literature Reviewmentioning

confidence: 99%

“…Petersen and Voigtlaender (2020). Moreover, CNN models have been mostly studied for handling high-dimensional data at input and not as output, as in He et al (2021). As a consequence, the available literature is left with a missing piece, which is to understand the approximation properties of convolutional layers when reconstructing functional signals.…”

Section: Literature Reviewmentioning

confidence: 99%

Approximation bounds for convolutional neural networks in operator learning

Franco¹,

Fresca²,

Manzoni³

et al. 2022

Preprint

View full text Add to dashboard Cite

Recently, deep Convolutional Neural Networks (CNNs) have proven to be successful when employed in areas such as reduced order modeling of parametrized PDEs. Despite their accuracy and efficiency, the approaches available in the literature still lack a rigorous justification on their mathematical foundations. Motivated by this fact, in this paper we derive rigorous error bounds for the approximation of nonlinear operators by means of CNN models. More precisely, we address the case in which an operator maps a finite dimensional input µ ∈ R p onto a functional output uµ : [0, 1] d → R, and a neural network model is used to approximate a discretized version of the input-to-output map. The resulting error estimates provide a clear interpretation of the hyperparameters defining the neural network architecture. All the proofs are constructive, and they ultimately reveal a deep connection between CNNs and the discrete Fourier transform. Finally, we complement the derived error bounds by numerical experiments that illustrate their application.

show abstract

“…An approximation theory for the 1-D DCNN with single channel induced by convolutional matrices like (4) have been extensively studied in a sequence of papers by Zhou et al [26,27,5,17]. Afterwards, Xu et al [9] extend to approximate L 2 functions by 2D CNNs with respect to odd filter size and stride one. It requires intermediate layers in the same dimension as the input through convolutions, which is slightly compatible with the practice.…”

Section: Letmentioning

confidence: 99%

Approximation analysis of CNNs from feature extraction view

Hong¹,

Li²

2022

Preprint

View full text Add to dashboard Cite

Deep learning based on deep neural networks has been very successful in many practical applications, but it lacks enough theoretical understanding due to the network architectures and structures. In this paper we establish the analysis for linear feature extraction by a deep multi-channel convolutional neural networks(CNNs), which demonstrates the power of deep learning over traditional linear transformations, like Fourier, Wavelets, Redundant dictionary coding methods. Moreover, we give an exact construction presenting how linear features extraction can be conducted efficiently with multi-channel CNNs. It can be applied to lower the essential dimension for approximating a high dimensional function. Rates of function approximation by such deep networks implemented with channels and followed by fully-connected layers are investigated as well. Harmonic analysis for factorizing linear features into multi-resolution convolutions plays an essential role in our work. Nevertheless, a dedicate vectorization of matrices is constructed, which bridges 1D CNN and 2D CNN and allows us have corresponding 2D analysis.

show abstract

Approximation Properties of Deep ReLU CNNs

Cited by 4 publications

References 30 publications

Solving Time-Dependent Parametric PDEs by Multiclass Classification-Based Reduced Order Model

Solving Time-Dependent Parametric PDEs by Multiclass Classification-Based Reduced Order Model

Approximation bounds for convolutional neural networks in operator learning

Approximation analysis of CNNs from feature extraction view

Contact Info

Product

Resources

About