A Mathematical Theory of Deep Convolutional Neural Networks for Feature Extraction

Discrete Deep Feature Extraction: A Theory and New Architectures

“…First steps towards addressing this question and developing a mathematical theory of DCNNs for feature extraction were made-for the continuous-time case-in (Mallat, 2012;Wiatowski & Bölcskei, 2015). Specifically, (Mallat, 2012) analyzed so-called scattering networks, where signals are propagated through layers that employ directional wavelet filters and modulus non-linearities but no intra-layer pooling.…”

“…The resulting wavelet-modulus feature extractor is horizontally (i.e., in every network layer) translation-invariant (accomplished by letting the wavelet scale parameter go to infinity) and deformationstable, both properties of significance in practical feature extraction applications. Recently, (Wiatowski & Bölcskei, 2015) considered Mallat-type networks with arbitrary filters (that may be learned or pre-specified), general Lipschitz-continuous non-linearities (e.g., rectified linear unit, shifted logistic sigmoid, hyperbolic tangent, and the modulus function), and a continuous-time pooling operator that amounts to a dilation. The essence of the results in (Wiatowski & Bölcskei, 2015) is that vertical (i.e., asymptotically in the network depth) translation invariance and Lipschitz continuity of the feature extractor are induced by the network structure per se rather than the specific choice of filters and non-linearities.…”

“…Recently, (Wiatowski & Bölcskei, 2015) considered Mallat-type networks with arbitrary filters (that may be learned or pre-specified), general Lipschitz-continuous non-linearities (e.g., rectified linear unit, shifted logistic sigmoid, hyperbolic tangent, and the modulus function), and a continuous-time pooling operator that amounts to a dilation. The essence of the results in (Wiatowski & Bölcskei, 2015) is that vertical (i.e., asymptotically in the network depth) translation invariance and Lipschitz continuity of the feature extractor are induced by the network structure per se rather than the specific choice of filters and non-linearities. For band-limited signals (Wiatowski & Bölcskei, 2015), cartoon functions (Grohs et al, 2016), and Lipschitz-continuous functions (Grohs et al, 2016), Lipschitz continuity of the feature extractor automatically leads to bounds on deformation sensitivity.…”

“…The essence of the results in (Wiatowski & Bölcskei, 2015) is that vertical (i.e., asymptotically in the network depth) translation invariance and Lipschitz continuity of the feature extractor are induced by the network structure per se rather than the specific choice of filters and non-linearities. For band-limited signals (Wiatowski & Bölcskei, 2015), cartoon functions (Grohs et al, 2016), and Lipschitz-continuous functions (Grohs et al, 2016), Lipschitz continuity of the feature extractor automatically leads to bounds on deformation sensitivity.…”

“…

First steps towards a mathematical theory of deep convolutional neural networks for feature extraction were made-for the continuous-time casein Mallat, 2012, andBölcskei, 2015. This paper considers the discrete case, introduces new convolutional neural network architectures, and proposes a mathematical framework for their analysis. Specifically, we establish deformation and translation sensitivity results of local and global nature, and we investigate how certain structural properties of the input signal are reflected in the corresponding feature vectors.

…”