On the $O(1/n)$ Convergence Rate of the Douglas–Rachford Alternating Direction Method

He, Bingsheng; Yuan, Xiaoming

doi:10.1137/110836936

Cited by 827 publications

(682 citation statements)

References 20 publications

Supporting

Mentioning

669

Contrasting

Unclassified

Order By: Relevance

“…It is known that in this case every limit point of the iterates is an optimal solution of the problem. The recent work of [21,22,26] have shown that, under some additional assumptions, the objective values generated by the ADMM algorithm and its accelerated version (which performs some additional line search steps for the dual update) converge at a rate of O(1/r) and O(1/r 2 ) respectively. Moreover, if the objective function f (x) is strongly convex and the constraint matrix E is row independent, then the ADMM is known to converge linearly to the unique minimizer of (1.1) [33].…”

Section: Alternating Direction Methods Of Multipliers (Admm)mentioning

confidence: 99%

On the linear convergence of the alternating direction method of multipliers

2016

View full text Add to dashboard Cite

We analyze the convergence rate of the alternating direction method of multipliers (ADMM) for minimizing the sum of two or more nonsmooth convex separable functions subject to linear constraints. Previous analysis of the ADMM typically assumes that the objective function is the sum of only two convex functions defined on two separable blocks of variables even though the algorithm works well in numerical experiments for three or more blocks. Moreover, there has been no rate of convergence analysis for the ADMM without strong convexity in the objective function. In this paper we establish the global linear convergence of the ADMM for minimizing the sum of any number of convex separable functions. This result settles a key question regarding the convergence of the ADMM when the number of blocks is more than two or if the strong convexity is absent. It also implies the linear convergence of the ADMM for several contemporary applications including LASSO, Group LASSO and Sparse Group LASSO without any strong convexity assumption. Our proof is based on estimating the distance from a dual feasible solution to the optimal dual solution set by the norm of a certain proximal residual, and by requiring the dual stepsize to be sufficiently small.KEY WORDS: Linear convergence, alternating directions of multipliers, error bound, dual ascent.

show abstract

Section: Alternating Direction Methods Of Multipliers (Admm)mentioning

confidence: 99%

On the linear convergence of the alternating direction method of multipliers

2016

View full text Add to dashboard Cite

show abstract

“…(13a) and (13b) (Eckstein and Bertsekas 1992). Recent theoretical analysis on convergence behavior of ADMM can also be found, for example, in He and Yuan (2012). Although a detailed theoretical analysis of the proposed method is beyond the scope of our current study, the basic characteristics can be understood from these general results.…”

Section: Stopping Criterion and Final Estimates Ofmentioning

confidence: 91%

Sparse and low-rank matrix regularization for learning time-varying Markov networks

2016

View full text Add to dashboard Cite

Statistical dependencies observed in real-world phenomena often change drastically with time. Graphical dependency models, such as the Markov networks (MNs), must deal with this temporal heterogeneity in order to draw meaningful conclusions about the transient nature of the target phenomena. However, in practice, the estimation of time-varying dependency graphs can be inefficient due to the potentially large number of parameters of interest. To overcome this problem, we propose such a novel approach to learning timevarying MNs that effectively reduces the number of parameters by constraining the rank of the parameter matrix. The underlying idea is that the effective dimensionality of the parameter space is relatively low in many realistic situations. Temporal smoothness and sparsity of the network are also incorporated as in previous studies. The proposed method is formulated as a convex minimization of a smoothed empirical loss with both the 1 -and the nuclearnorm regularization terms. This non-smooth optimization problem is numerically solved by the alternating direction method of multipliers. We take the Ising model as a fundamental example of an MN, and we show in several simulation studies that the rank-reducing effect from the nuclear norm can improve the estimation performance of time-varying dependency graphs. We also demonstrate the utility of the method for analyzing real-world datasets for improving the interpretability and predictability of the obtained networks.

show abstract

“…For γ > 0 and μ ∈ [0, 2], the iterates α k converge to the minimizer of equation (2.5). For more details on the convergence of the Douglas-Rachford algorithm, see [34,35]. The coefficients identified by equation (2.5), denoted α * , will typically have some bias due to the shrinkage function.…”

Section: (B) Numerical Methods and Algorithmmentioning

confidence: 99%

Learning partial differential equations via data discovery and sparse optimization

Schaeffer¹

2017

Proc. R. Soc. A.

374

325

View full text Add to dashboard Cite

We investigate the problem of learning an evolution equation directly from some given data. This work develops a learning algorithm to identify the terms in the underlying partial differential equations and to approximate the coefficients of the terms only using data. The algorithm uses sparse optimization in order to perform feature selection and parameter estimation. The features are data driven in the sense that they are constructed using nonlinear algebraic equations on the spatial derivatives of the data. Several numerical experiments show the proposed method's robustness to data noise and size, its ability to capture the true features of the data, and its capability of performing additional analytics. Examples include shock equations, pattern formation, fluid flow and turbulence, and oscillatory convection.

show abstract

On the $O(1/n)$ Convergence Rate of the Douglas–Rachford Alternating Direction Method

Cited by 827 publications

References 20 publications

On the linear convergence of the alternating direction method of multipliers

On the linear convergence of the alternating direction method of multipliers

Sparse and low-rank matrix regularization for learning time-varying Markov networks

Learning partial differential equations via data discovery and sparse optimization

Contact Info

Product

Resources

About