A Mathematical Theory of Deep Convolutional Neural Networks for Feature Extraction

Abstract

Deep convolutional neural networks have led to breakthrough results in numerous practical machine learning tasks such as classification of images in the ImageNet data set, control-policy-learning to play Atari games or the board game Go, and image captioning. Many of these applications first perform feature extraction and then feed the results thereof into a trainable classifier. The mathematical analysis of deep convolutional neural networks for feature extraction was initiated by Mallat, 2012. Specifically, Mallat considered so-called scattering networks based on a wavelet transform followed by the modulus non-linearity in each network layer, and proved translation invariance (asymptotically in the wavelet scale parameter) and deformation stability of the corresponding feature extractor. This paper complements Mallat's results by developing a theory that encompasses general convolutional transforms, or in more technical parlance, general semi-discrete frames (including Weyl-Heisenberg filters, curvelets, shearlets, ridgelets, wavelets, and learned filters), general Lipschitz-continuous non-linearities (e.g., rectified linear units, shifted logistic sigmoids, hyperbolic tangents, and modulus functions), and general Lipschitz-continuous pooling operators emulating, e.g., sub-sampling and averaging. In addition, all of these elements can be different in different network layers. For the resulting feature extractor we prove a translation invariance result of vertical nature in the sense of the features becoming progressively more translation-invariant with increasing network depth, and we establish deformation sensitivity bounds that apply to signal classes such as, e.g., bandlimited functions, cartoon functions, and Lipschitz functions.

Feasibility of deep learning for predicting live birth from a blastocyst image in patients classified by age

“…The profiles of the accuracies, with standard deviation (SD), are shown according to the number of the training dataset, as shown in Figure 2. The best numbers obtained in the training dataset were 17112, 7848, 8085, and 12 144 in the age groups 35 Table 3. The rectified linear unit function was the best among the logistic sigmoid function, the hyperbolic tangent function, and the Heaviside theta function (data not shown).…”

“…We developed classifier programs in each age category using supervised deep learning with a convolutional neural network 30,31 that tried to mimic the visual cortex of the mammal brain 21,23,[32][33][34][35] and used L2 regularization 36,37 to categorize blastocyst images as either in the live birth or the nonlive birth category and to obtain the mathematical probability for predicting each category. We performed deep learning with a convolutional neural network with eleven layers consisting of a combination of convolution layers with varying output channels and kernel sizes, 38,39 pooling layers, 41,42 flattened layers, 45 linear layers, 46,47 rectified linear unit layers, 48,49 and a softmax layer 50,51 that demonstrated the probability of a live birth from an image of the blastocyst.…”

“…As for a sample, the histogram for live birth and nonlive birth in patients aged [35][36][37] years is demonstrated in the upper graph shown in Figure 3. These data were transformed to obtain the incidence of live birth and then fitted to a logistic regression model, as shown in the lower graph in Figure 3.…”

Cloud‐aided online EEG classification system for brain healthcare: A case study of depression evaluation with a lightweight CNN

“…It enables dimensionality reduction and supports effective exploration of the signals of interest, which determines the performance of classification. 14 The common spatial pattern achieves the highest accuracy of 87.4%. 15 For matrix factorization methods, the sparse nonnegative matrix factorization achieves the best accuracy of 86.61%, superior to those of ICA, PCA, NMF, Wavelets, and EMD.…”

“…Basically, there exists no general solution to reliable EEG denoising. In the context of conventional EEG classification, denoising of high quality is a prerequisite for feature extraction . It enables dimensionality reduction and supports effective exploration of the signals of interest, which determines the performance of classification . The common spatial pattern achieves the highest accuracy of 87.4% .…”