Convergence of Deep ReLU Networks

Abstract

We explore convergence of deep neural networks with the popular ReLU activation function, as the depth of the networks tends to infinity. To this end, we introduce the notion of activation domains and activation matrices of a ReLU network. By replacing applications of the ReLU activation function by multiplications with activation matrices on activation domains, we obtain an explicit expression of the ReLU network. We then identify the convergence of the ReLU networks as convergence of a class of infinite products of matrices. Sufficient and necessary conditions for convergence of these infinite products of matrices are studied. As a result, we establish necessary conditions for ReLU networks to converge that the sequence of weight matrices converges to the identity matrix and the sequence of the bias vectors converges to zero as the depth of ReLU networks increases to infinity. Moreover, we obtain sufficient conditions in terms of the weight matrices and bias vectors at hidden layers for pointwise convergence of deep ReLU networks. These results provide mathematical insights to the design strategy of the well-known deep residual networks in image classification.

Convergence of Deep Convolutional Neural Networks

“…Having the above matrix form, conditions on the masks and bias vectors that ensure convergence of the convolutional neural network (2.1) as the depth n tends to infinity are now reformulated as conditions on the weight matrices and the bias vectors. Convergence of deep ReLU neural networks with a fixed width was recently studied in [21]. The matrix form for CNNs differs from that for DNNs considered in [21] in that the widths in CNNs are increasing while those in [21] are fixed.…”

“…In particular, the approximation and expressive powers of deep neural networks have been a focus. A brief review of the literature for this aspect was contained in the introduction of [21]. We also refer readers to the two recent surveys [4,6] for further details.…”

“…We believe that this question is fundamental in mathematical theory of deep learning. In the recent work [21], we studied this question for deep ReLU networks with a fixed width and were able to provide a mathematical interpretation to the design of the successful deep residual networks [9] based on the obtained results.…”

“…However, the study [21] and those studies [1,3,4,5,6,12,13,14,15,16,17,22] on approximation and expressive powers of deep neural networks were all about fully connected networks without prespecified structures, and thus did not cover the important convolutional neural networks (CNNs). CNNs use filter masks to perform convolution with the input data at each layer and are particularly useful and efficient in extracting features of speech and image data.…”

“…For such CNNs, the widths are not fixed but are increasing to infinity. This was not considered in [21] where the width of neural networks is fixed. There are some major difficulties caused by the increasing widths.…”