An inertial Bregman generalized alternating direction method of multipliers for nonconvex optimization

Xu, Jie; Chao, Miantao

doi:10.1007/s12190-021-01590-1

Cited by 15 publications

(5 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…where ∂g (•) represents the Clarke subgradient of g(•), as defined in Definition 1. Combining (7) with…”

Section: A the Update Rule Of Ymentioning

confidence: 99%

“…Lemma 1 Let w k := (x k , y k , λ k ) be the iterate satisfying the conditions (7) and (11). Suppose Assumption 1 (a), (b), (c) hold and AL function is bounded below, we can choose the parameters in Algorithm 1 such that…”

Section: A Global Convergence and Sublinear Convergencementioning

confidence: 99%

“…Matrices A ∈ R m×nx and B ∈ R m×ny serve as linear operators encoding the model structure. Problem (1) covers a broad range of applications in statistical learning, compressive sensing, and machine learning [3], [4], especially in neural networks, such as the graph-guided fused lasso [5] and the nonconvex problem with SCAD penalty [6], [7].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

An Accelerated Variance Reduction Stochastic ADMM for Nonconvex Nonsmooth Optimization

Zeng

Wang

Bai

et al. 2022

2022 China Automation Congress (CAC)

View full text Add to dashboard Cite

The nonconvex and nonsmooth finite-sum optimization problem with linear constraint has attracted much attention in the fields of artificial intelligence, computer, and mathematics, due to its wide applications in machine learning and the lack of efficient algorithms with convincing convergence theories. A popular approach to solve it is the stochastic Alternating Direction Method of Multipliers (ADMM), but most stochastic ADMM-type methods focus on convex models. In addition, the variance reduction (VR) and acceleration techniques are useful tools in the development of stochastic methods due to their simplicity and practicability in providing acceleration characteristics of various machine learning models. However, it remains unclear whether accelerated SVRG-ADMM algorithm (ASVRG-ADMM), which extends SVRG-ADMM by incorporating momentum techniques, exhibits a comparable acceleration characteristic or convergence rate in the nonconvex setting. To fill this gap, we consider a general nonconvex nonsmooth optimization problem and study the convergence of ASVRG-ADMM. By utilizing a well-defined potential energy function, we establish its sublinear convergence rate O(1/T ), where T denotes the iteration number. Furthermore, under the additional Kurdyka-Lojasiewicz (KL) property which is less stringent than the frequently used conditions for showcasing linear convergence rates, such as strong convexity, we show that the ASVRG-ADMM sequence almost surely has a finite length and converges to a stationary solution with a linear convergence rate. Several experiments on solving the graph-guided fused lasso problem and regularized logistic regression problem validate that the proposed ASVRG-ADMM performs better than the state-of-the-art methods.

show abstract

“…where ∂g (•) represents the Clarke subgradient of g(•), as defined in Definition 1. Combining (7) with…”

Section: A the Update Rule Of Ymentioning

confidence: 99%

Section: A Global Convergence and Sublinear Convergencementioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

An Accelerated Variance Reduction Stochastic ADMM for Nonconvex Nonsmooth Optimization

Zeng

Wang

Bai

et al. 2022

2022 China Automation Congress (CAC)

View full text Add to dashboard Cite

show abstract

“…Recently, it has been frequently applied to the minimization of nonconvex functions. In [39,40], the inertia term was introduced into the nonconvex optimization problem to improve the convergence speed. Refs.…”

Section: Inertia Itemmentioning

confidence: 99%

IPGM: Inertial Proximal Gradient Method for Convolutional Dictionary Learning

Wei

Wang

et al. 2021

Electronics

View full text Add to dashboard Cite

Inspired by the recent success of the proximal gradient method (PGM) and recent efforts to develop an inertial algorithm, we propose an inertial PGM (IPGM) for convolutional dictionary learning (CDL) by jointly optimizing both an ℓ2-norm data fidelity term and a sparsity term that enforces an ℓ1 penalty. Contrary to other CDL methods, in the proposed approach, the dictionary and needles are updated with an inertial force by the PGM. We obtain a novel derivative formula for the needles and dictionary with respect to the data fidelity term. At the same time, a gradient descent step is designed to add an inertial term. The proximal operation uses the thresholding operation for needles and projects the dictionary to a unit-norm sphere. We prove the convergence property of the proposed IPGM algorithm in a backtracking case. Simulation results show that the proposed IPGM achieves better performance than the PGM and slice-based methods that possess the same structure and are optimized using the alternating-direction method of multipliers (ADMM).

show abstract

“…Moreover, inertial technique has also been applied to ADMM for solving convex optimization problems (Bot et al 2014;, and nonconvex problems (Chao et al 2020;Xu et al 2021). Especially, Sun (2019) proposed inertial style ADMM (iADMM) for the image deblurring, its iteration form is as follows…”

Section: Introductionmentioning

confidence: 99%

A Framework of Inertial Regularized ADMM for Separable Nonconvex Optimization Problems

Chao

Geng

Zhao

2023

Preprint

Self Cite

View full text Add to dashboard Cite

The alternating direction method of multipliers (ADMM) is an effective algorithm for solving optimization problems with separable structures. Recently, inertial technique has been widely used in various algorithms to accelerate its convergence speed and enhance the numerical performance. There are a lot of convergence analyses for solving the convex optimization problems by combining inertial technique with ADMM, while the research on the nonconvex cases is still in its infancy. In this paper, we propose an algorithm framework of inertial regularized ADMM (iRADMM) for a class of two-block nonconvex optimization problems. Under some assumptions, we establish the global and strong convergence of the proposed method. Furthermore, we apply the iRADMM to solve the signal recovery, image reconstruction and SCAD penalty problem. The numerical results demonstrate the efficiency of the iRADMM algorithm and also illustrate the effectiveness of the introduced inertial term.

show abstract

An inertial Bregman generalized alternating direction method of multipliers for nonconvex optimization

Cited by 15 publications

References 40 publications

An Accelerated Variance Reduction Stochastic ADMM for Nonconvex Nonsmooth Optimization

An Accelerated Variance Reduction Stochastic ADMM for Nonconvex Nonsmooth Optimization

IPGM: Inertial Proximal Gradient Method for Convolutional Dictionary Learning

A Framework of Inertial Regularized ADMM for Separable Nonconvex Optimization Problems

Contact Info

Product

Resources

About