Sparseness Analysis in the Pretraining of Deep Neural Networks

Abstract

A major progress in deep multilayer neural networks (DNNs) is the invention of various unsupervised pretraining methods to initialize network parameters which lead to good prediction accuracy. This paper presents the sparseness analysis on the hidden unit in the pretraining process. In particular, we use the L -norm to measure sparseness and provide some sufficient conditions for that pretraining leads to sparseness with respect to the popular pretraining models-such as denoising autoencoders (DAEs) and restricted Boltzmann machines (RBMs). Our experimental results demonstrate that when the sufficient conditions are satisfied, the pretraining models lead to sparseness. Our experiments also reveal that when using the sigmoid activation functions, pretraining plays an important sparseness role in DNNs with sigmoid (Dsigm), and when using the rectifier linear unit (ReLU) activation functions, pretraining becomes less effective for DNNs with ReLU (Drelu). Luckily, Drelu can reach a higher recognition accuracy than DNNs with pretraining (DAEs and RBMs), as it can capture the main benefit (such as sparseness-encouraging) of pretraining in Dsigm. However, ReLU is not adapted to the different firing rates in biological neurons, because the firing rate actually changes along with the varying membrane resistances. To address this problem, we further propose a family of rectifier piecewise linear units (RePLUs) to fit the different firing rates. The experimental results show that the performance of RePLU is better than ReLU, and is comparable with those with some pretraining techniques, such as RBMs and DAEs.

Fashion Style Generator

“…For patch style loss network ϕ s , since existing networks are mainly trained for full images, instead of directly applying the existing pretrained discriminator network, we apply Generative Adversarial Network (GAN) [Goodfellow et al, 2014;Radford et al, 2015] for learning ϕ s and meanwhile initializing the parameters of decoder D e of the generator. We will describe it in the next subsection.…”

“…We firstly describe utilizing GAN [Goodfellow et al, 2014;Radford et al, 2015] for learning patch style network ϕ s and meanwhile initializing the parameters of decoder D e . The inputs of this stage are image patches X (2) and the style image y s .…”

“…Applying artificial intelligence to solve problems in art and fashion fields attract a lot of attentions such as fashion style classification [FYihui Ma and Tong, 2017;Jiang et al, 2016a], clothing parsing [Yamaguchi et al, 2013;Yamaguchi et al, 2012], clothing retrieval [Jiang et al, 2016b] and recommendation [Fu12 et al, 2017]. In this paper, we focus on a novel problem: fashion style generation.…”

Regularized deep learning for fashion and social media data analysis

“…Existing methods have shown satisfactory performance of visual feature representation using deep learning methods [46,82,83,84,16,85]. For example, Kiapour et al applied CNN features as image representation, and calculated the cross-entropy loss measuring whether two images are matched or non-matched.…”

“…Therefore, we utilize a pretrained image classification network (i.e., VGG-19) [112,113] as the initialization of E n . Also, the VGG network is utilized as the global loss network φ and the patch content loss network ϕ c .…”