On random weights for texture generation in one layer CNNS

Abstract

Recent work in the literature has shown experimentally that one can use the lower layers of a trained convolutional neural network (CNN) to model natural textures. More interestingly, it has also been experimentally shown that only one layer with random filters can also model textures although with less variability. In this paper we ask the question as to why one layer CNNs with random filters are so effective in generating textures? We theoretically show that one layer convolutional architectures (without a non-linearity) paired with the an energy function used in previous literature, can in fact preserve and modulate frequency coefficients in a manner so that random weights and pretrained weights will generate the same type of images. Based on the results of this analysis we question whether similar properties hold in the case where one uses one convolution layer with a non-linearity. We show that in the case of ReLu non-linearity there are situations where only one input will give the minimum possible energy whereas in the case of no nonlinearity, there are always infinite solutions that will give the minimum possible energy. Thus we can show that in certain situations adding a ReLu non-linearity generates less variable images.

Design of Motion Damage Estimation Model Based on Improved Recursive Neural Network Algorithm

“…The prediction output of RNN at each moment is composed of two parts, one is from the input of the current moment, and the other is from the hidden state of the last moment. At present, RNN shows great advantages in speech recognition, machine translation, text language processing, and other fields [14,15].…”

Generic Perceptual Loss for Modeling Structured Output Dependencies

“…They found that during optimization for a style transfer task, the perceptual loss with a proper weight scale can work well with untrained, randomly-weighted networks, and generate competitive results as prior work [8] with pretrained weights. Mongia et al [19] prove why one-layer CNNs with random weights can successfully generate textures. The randomly-weighted networks are also employed in unsupervised learning [29] and reinforcement learning [7].…”

Dynamic Texture Recognition Using 3D Random Features

“…In the field of deep learning, networks with random weights have also shown preferable properties [27,28,29]. Giryes et al [27] found networks with random Gaussian filters can perform distance-preserving embedding of the original data while training is to find hyperplanes to separate different data points.…”

“…Gilbert et al [28] connected networks with random Gaussian filters for compressed sensing and analyzed the invertibility of networks, in which they observed that networks with random Gaussian filters could achieve good classification performance. Mongia et al [29] applied random weights for static texture generation.…”