Structured Sparsity of Convolutional Neural Networks via Nonconvex Sparse Group Regularization

Bui, Kevin; Park, Fredrick; Zhang, Shuai; Qi, Yingyong; Xin, Jack

doi:10.3389/fams.2020.529564

Cited by 11 publications

(4 citation statements)

References 85 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, in the context of bi‐level selection procedures, alternative combinations have never been systematically investigated, so its full potential has not yet been exploited (Buch et al., 2023). More precisely, the limited approaches were studied independently (Bui et al., 2021; Liu et al., 2013), so their relationship was not formalized, and they were compared neither with each other nor with approaches of the hierarchical framework.…”

Section: Introductionmentioning

confidence: 99%

Sparse Group Penalties for bi‐level variable selection

Buch,

Schulz,

Schmidtmann

et al. 2024

Biometrical J

View full text Add to dashboard Cite

Many data sets exhibit a natural group structure due to contextual similarities or high correlations of variables, such as lipid markers that are interrelated based on biochemical principles. Knowledge of such groupings can be used through bi‐level selection methods to identify relevant feature groups and highlight their predictive members. One of the best known approaches of this kind combines the classical Least Absolute Shrinkage and Selection Operator (LASSO) with the Group LASSO, resulting in the Sparse Group LASSO. We propose the Sparse Group Penalty (SGP) framework, which allows for a flexible combination of different SGL‐style shrinkage conditions. Analogous to SGL, we investigated the combination of the Smoothly Clipped Absolute Deviation (SCAD), the Minimax Concave Penalty (MCP) and the Exponential Penalty (EP) with their group versions, resulting in the Sparse Group SCAD, the Sparse Group MCP, and the novel Sparse Group EP (SGE). Those shrinkage operators provide refined control of the effect of group formation on the selection process through a tuning parameter. In simulation studies, SGPs were compared with other bi‐level selection methods (Group Bridge, composite MCP, and Group Exponential LASSO) for variable and group selection evaluated with the Matthews correlation coefficient. We demonstrated the advantages of the new SGE in identifying parsimonious models, but also identified scenarios that highlight the limitations of the approach. The performance of the techniques was further investigated in a real‐world use case for the selection of regulated lipids in a randomized clinical trial.

show abstract

Section: Introductionmentioning

confidence: 99%

Sparse Group Penalties for bi‐level variable selection

Buch,

Schulz,

Schmidtmann

et al. 2024

Biometrical J

View full text Add to dashboard Cite

show abstract

“…Moreover, Wen et al (2016) show that by using the gradient descent methods, less training time is required by DNN with a group sparse regularizer compared with that required by DNN with a lasso regularizer. The group sparse regularizer also appears in convolution neural network (Bui et al, 2021) and other machine learning problems (Meier et al, 2008;Jenatton et al, 2011;Simon et al, 2013), etc. Hence, we focus on training the leaky ReLU network with the l 2,1 regularizer for pursuing the group sparsity.…”

Section: Introductionmentioning

confidence: 99%

An Inexact Augmented Lagrangian Algorithm for Training Leaky ReLU Neural Network with Group Sparsity

Liu¹,

Liu²,

Chen³

2022

Preprint

View full text Add to dashboard Cite

The leaky ReLU network with a group sparse regularization term has been widely used in the recent years. However, training such network yields a nonsmooth nonconvex optimization problem and there exists a lack of approaches to compute a stationary point deterministically. In this paper, we first resolve the multi-layer composite term in the original optimization problem by introducing auxiliary variables and additional constraints. We show the new model has a nonempty and bounded solution set and its feasible set satisfies the Mangasarian-Fromovitz constraint qualification. Moreover, we show the relationship between the new model and the original problem. Remarkably, we propose an inexact augmented Lagrangian algorithm for solving the new model, and show the convergence of the algorithm to a KKT point. Numerical experiments demonstrate that our algorithm is more efficient for training sparse leaky ReLU neural networks than some well-known algorithms.

show abstract

“…Both Ma et al [47] and Pandit et al [48] proposed a regularizer that combines group sparsity and T 1 and applied it to CNNs for image classification. Bui et al [49] generalized sparse group lasso to incorporate nonconvex regularizers and applied it to various CNN architectures. Li et al [50] introduced sparsity-inducing matrices into CNNs and imposed group sparsity on the rows or columns via 1 or other nonconvex regularizers to prune filters and/or channels.…”

Section: Introductionmentioning

confidence: 99%

Improving Network Slimming With Nonconvex Regularization

et al. 2021

Self Cite

View full text Add to dashboard Cite

Convolutional neural networks (CNNs) have developed to become powerful models for various computer vision tasks ranging from object detection to semantic segmentation. However, most of the state-of-the-art CNNs cannot be deployed directly on edge devices such as smartphones and drones, which need low latency under limited power and memory bandwidth. One popular, straightforward approach to compressing CNNs is network slimming, which imposes 1 regularization on the channel-associated scaling factors via the batch normalization layers during training. Network slimming thereby identifies insignificant channels that can be pruned for inference. In this paper, we propose replacing the penalty with an alternative nonconvex, sparsity-inducing penalty in order to yield a more compressed and/or accurate CNN architecture. We investigate p (0 < p < 1), transformed 1 (T 1 ), minimax concave penalty (MCP), and smoothly clipped absolute deviation (SCAD) due to their recent successes and popularity in solving sparse optimization problems, such as compressed sensing and variable selection. We demonstrate the effectiveness of network slimming with nonconvex penalties on VGGNet, Densenet, and Resnet on standard image classification datasets. Based on the numerical experiments, T 1 preserves model accuracy against channel pruning, 1/2,3/4 yield better compressed models with similar accuracies after retraining as 1 , and MCP and SCAD provide more accurate models after retraining with similar compression as 1 . Network slimming with T 1 regularization also outperforms the latest Bayesian modification of network slimming in compressing a CNN architecture in terms of memory storage while preserving its model accuracy after channel pruning.INDEX TERMS Convolutional neural networks (CNN), machine learning, deep learning, network pruning, nonconvex optimization

show abstract

Structured Sparsity of Convolutional Neural Networks via Nonconvex Sparse Group Regularization

Cited by 11 publications

References 85 publications

Sparse Group Penalties for bi‐level variable selection

Sparse Group Penalties for bi‐level variable selection

An Inexact Augmented Lagrangian Algorithm for Training Leaky ReLU Neural Network with Group Sparsity

Improving Network Slimming With Nonconvex Regularization

Contact Info

Product

Resources

About