Guided Dropout

Abstract

Dropout is often used in deep neural networks to prevent over-fitting. Conventionally, dropout training invokes random drop of nodes from the hidden layers of a Neural Network. It is our hypothesis that a guided selection of nodes for intelligent dropout can lead to better generalization as compared to the traditional dropout. In this research, we propose "guided dropout" for training deep neural network which drop nodes by measuring the strength of each node. We also demonstrate that conventional dropout is a specific case of the proposed guided dropout. Experimental evaluation on multiple datasets including MNIST, CIFAR10, CIFAR100, SVHN, and Tiny ImageNet demonstrate the efficacy of the proposed guided dropout.

Maximum Relevance Minimum Redundancy Dropout with Informative Kernel Determinantal Point Process

“…To verify the effectiveness of our proposed algorithms, we compare our approaches with recent dropout techniques, including the Automatic dropout [ 17 ], Controlled dropout [ 22 ], DropMI dropout [ 24 ], Guided dropout [ 15 ], Concrete dropout [ 23 ], and Targeted dropout [ 18 ], as well as the Standard dropout [ 10 ]. All the experiments are carried out using GPU-based Tensorflow [ 44 ] on Python 3.…”

“…The Standard dropout removes each computational latent unit using a fixed removal probability p independent of the rest of latent units. In recent studies, a variety of methods such as Standout [ 14 ], Guided dropout [ 15 ], Adversarial dropout [ 16 ], Automatic dropout [ 17 ], and Targeted dropout [ 18 ] etc. are proposed to achieve a more semantic dropout mechanism.…”

“…Controlled dropout [ 22 ] is a more memory efficient and faster version of standard dropout that the authors have suggested to gather and relocate non-zero weights in a new memory. Guided dropout [ 15 ] and Concrete dropout [ 23 ] are two efficient approaches that seek to find the p parameter by minimizing a defined objective function. By making use of the MI concept, Chen et al [ 24 ], proposed DropMI method that selects most relevant neurons to the target vector.…”

Group-Wise Dynamic Dropout Based on Latent Semantic Variations

“…However, the side-model based approaches introduce significant computation and memory overhead. (Keshari, Singh, and Vatsa 2019) proposed the guided dropout to drop network nodes with high strength to encourage low strength nodes. (Wang, Zhou, and Bilmes 2019) proposed to Jumpout samples the dropout probability from a monotone decreasing distribution (e.g., the right half of a Gaussian) such that each linear piece of the network can learn better for data points from nearby than more distant regions to improve generalization of DNNs with ReLU activations.…”

“…In particular, it was shown that the generalization ability can be improved by dropping nodes selectively based on some prior knowledge of the network. For instance, (Keshari, Singh, and Vatsa 2019) learns a strength parameter by stochastic gradient descent (SGD) of the network for guiding dropout regularization of each node. (Wang, Zhou, and Bilmes 2019) adapts the dropout probability by normalizing it at each layer and every training batch such that the effective dropping rate on those activated units is kept the same during the training.…”