Efficient Approximate Solution Path Algorithm for Order Weight L_1-Norm with Accuracy Guarantee

Bao, Runxue; Gu, Bin; Huang, Heng

doi:10.1109/icdm.2019.00107

Cited by 6 publications

(4 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our goal in this paper is to propose an efficient extra proximal gradient algorithm that employs the Nesterov’s acceleration technique and the extra gradient scheme, and unroll this algorithm into a deep neural network called the extra proximal gradient network (EPGN) to solve a class of inverse problems ( 1 ). Motivated by the least absolute shrinkage and selection operator (LASSO) [ 11 , 12 , 13 ], our EPGN implicitly adopts an

-type regularization in ( 1 ) with a nonlinear sparsification mapping learned from data. The proximal operator of this regularization is elaborated by several linear convolutions, nonlinear activation functions, and shrinkage operations for robust sparse feature selection in EPGN.…”

Section: Introductionmentioning

confidence: 99%

Extra Proximal-Gradient Network with Learned Regularization for Image Compressive Sensing Reconstruction

Zhang

Ye²,

Chen³

2022

J. Imaging

View full text Add to dashboard Cite

Learned optimization algorithms are promising approaches to inverse problems by leveraging advanced numerical optimization schemes and deep neural network techniques in machine learning. In this paper, we propose a novel deep neural network architecture imitating an extra proximal gradient algorithm to solve a general class of inverse problems with a focus on applications in image reconstruction. The proposed network features learned regularization that incorporates adaptive sparsification mappings, robust shrinkage selections, and nonlocal operators to improve solution quality. Numerical results demonstrate the improved efficiency and accuracy of the proposed network over several state-of-the-art methods on a variety of test problems.

show abstract

Section: Introductionmentioning

confidence: 99%

Extra Proximal-Gradient Network with Learned Regularization for Image Compressive Sensing Reconstruction

Zhang

Ye²,

Chen³

2022

J. Imaging

View full text Add to dashboard Cite

show abstract

“…However, manually tunning hyperparameters is often timeconsuming, and depends heavily on human's prior knowledge, which makes it difficult to find the optimal hyperparameters. Therefore, to choose the set hyperparameters automatically has attracted great attention, and large amounts of HO methods have emerged, such as grid search, random search (Bergstra et al 2011;Bergstra and Bengio 2012), solution path (Gu and Sheng 2017;Gu, Liu, and Huang 2017;Bao, Gu, and Huang 2019;Gu and Ling 2015), and several Bayesian methods (Thornton et al 2013;Brochu, Cora, and De Freitas 2010;Swersky, Snoek, and Adams 2014;Wu et al 2019).…”

Section: Introductionmentioning

confidence: 99%

Improved Penalty Method via Doubly Stochastic Gradients for Bilevel Hyperparameter Optimization

Shi

2021

AAAI

Self Cite

View full text Add to dashboard Cite

Hyperparameter optimization (HO) is an important problem in machine learning which is normally formulated as a bilevel optimization problem. Gradient-based methods are dominant in bilevel optimization due to their high scalability to the number of hyperparameters, especially in a deep learning problem. However, traditional gradient-based bilevel optimization methods need intermediate steps to obtain the exact or approximate gradient of hyperparameters, namely hypergradient, for the upper-level objective, whose complexity is high especially for high dimensional datasets. Recently, a penalty method has been proposed to avoid the computation of the hypergradient, which speeds up the gradient-based BHO methods. However, the penalty method may result in a very large number of constraints, which greatly limits the efficiency of this method, especially for high dimensional data problems. To address this limitation, in this paper, we propose a doubly stochastic gradient descent algorithm (DSGPHO) to improve the efficiency of the penalty method. Importantly, we not only prove the proposed method can converge to the KKT condition of the original problem in a convex setting, but also provide the convergence rate of DSGPHO which is the first result in the references of gradient-based bilevel optimization as far as we know. We compare our method with three state-of-the-art gradient-based methods in three tasks, i.e., data denoising, few-shot learning, and training data poisoning, using several large-scale benchmark datasets. All the results demonstrate that our method outperforms or is comparable to the existing methods in terms of accuracy and efficiency.

show abstract

“…OWL regression (Bogdan et al, 2013;Zeng & Figueiredo, 2014;Bogdan et al, 2015;Figueiredo & Nowak, 2016;Bao et al, 2019) has emerged as a useful procedure for high-dimensional sparse regression recently, which can promote the sparsity and grouping simultaneously. Unlike group Lasso (Yuan & Lin, 2006) and its variants, OWL regression can identify precise grouping structures of strongly correlated covariates automatically during the learning process without any prior information of feature groups.…”

Section: Introductionmentioning

confidence: 99%

Fast OSCAR and OWL Regression via Safe Screening Rules

Bao¹,

Gu²,

Huang³

2020

Preprint

Self Cite

View full text Add to dashboard Cite

Ordered Weighted L 1 (OWL) regularized regression is a new regression analysis for highdimensional sparse learning. Proximal gradient methods are used as standard approaches to solve OWL regression. However, it is still a burning issue to solve OWL regression due to considerable computational cost and memory usage when the feature or sample size is large. In this paper, we propose the first safe screening rule for OWL regression by exploring the order of the primal solution with the unknown order structure via an iterative strategy, which overcomes the difficulties of tackling the non-separable regularizer. It effectively avoids the updates of the parameters whose coefficients must be zero during the learning process. More importantly, the proposed screening rule can be easily applied to standard and stochastic proximal gradient methods. Moreover, we prove that the algorithms with our screening rule are guaranteed to have identical results with the original algorithms. Experimental results on a variety of datasets show that our screening rule leads to a significant computational gain without any loss of accuracy, compared to existing competitive algorithms.

show abstract

Efficient Approximate Solution Path Algorithm for Order Weight L_1-Norm with Accuracy Guarantee

Cited by 6 publications

References 18 publications

Extra Proximal-Gradient Network with Learned Regularization for Image Compressive Sensing Reconstruction

Extra Proximal-Gradient Network with Learned Regularization for Image Compressive Sensing Reconstruction

Improved Penalty Method via Doubly Stochastic Gradients for Bilevel Hyperparameter Optimization

Fast OSCAR and OWL Regression via Safe Screening Rules

Contact Info

Product

Resources

About