Group sparse regularization for deep neural networks

Abstract

In this paper, we address the challenging task of simultaneously optimizing (i) the weights of a neural network, (ii) the number of neurons for each hidden layer, and (iii) the subset of active input features (i.e., feature selection). While these problems are traditionally dealt with separately, we propose an efficient regularized formulation enabling their simultaneous parallel execution, using standard optimization routines. Specifically, we extend the group Lasso penalty, originally proposed in the linear regression literature, to impose group-level sparsity on the network's connections, where each group is defined as the set of outgoing weights from a unit. Depending on the specific case, the weights can be related to an input variable, to a hidden neuron, or to a bias unit, thus performing simultaneously all the aforementioned tasks in order to obtain a compact network. We carry out an extensive experimental evaluation, in comparison with classical weight decay and Lasso penalties, both on a toy dataset for handwritten digit recognition, and multiple realistic mid-scale classification benchmarks. Comparative results demonstrate the potential of our proposed sparse group Lasso penalty in producing extremely compact networks, with a significantly lower number of input features, with a classification accuracy which is equal or only slightly inferior to standard regularization terms

ROBO: Robust, Fully Neural Object Detection for Robot Soccer

“…The most important change of our training method is the application of L1 regularization to the weights [23]. While L1 regularization is primarily a way to avoid overfitting, it has a desirable side effect: sparsifying the network's weight matrices.…”

“…While L1 regularization is primarily a way to avoid overfitting, it has a desirable side effect: sparsifying the network's weight matrices. While most state-of-the-art methods of pruning devote considerable computational expense to find the least influential weights, using L1 regularization ensures that the majority of the weights are already effectively zero and can be pruned without affecting the network at all [23].…”

SSN: Learning Sparse Switchable Normalization via SparsestMax

“…Other work focusing on the sparsity of parameters in DNN is also related to this article. In [22], group Lasso penalty is adopted to impose group-level sparsity on networks connections. But this work can hardly satisfy our standardization constraints, i.e.…”

VACL: Variance-Aware Cross-Layer Regularization for Pruning Deep Residual Networks

“…In this case, a pruned model can take advantage of dense matrix computation just as the original model does. L 1 norm regularization is commonly used to enforce individual parameter sparsity, and Group Lasso [39] is normally used to enforce structural parameter sparsity [32].…”