Derivation and Analysis of the Primal-Dual Method of Multipliers Based on Monotone Operator Theory

Sherson, Thomas; Heusdens, Richard; Kleijn, W. Bastiaan

doi:10.48550/arxiv.1706.02654

Cited by 3 publications

(8 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Convergence analyses for synchronous PDMM have been provided in [13] and [21]. We present one convergence property from [13] in a theorem below.…”

Section: A Synchronous Pdmmmentioning

confidence: 99%

“…Remark 1. We note that the work in [21] focuses on a special structure of the matrix set P, namely P i,j = ρI for every P i,j ∈ P, where ρ > 0. One research direction is to extend the work of [21] to the general matrix form P i,j ≻ 0, (i, j) ∈ E.…”

Section: A Synchronous Pdmmmentioning

confidence: 99%

“…Differently from synchronous PDMM, the theoretical analysis for asynchronous PDMM depends on how the individual node is selected per iteration. [13] provides convergence analysis for the cyclic updating scheme while [21] utlilizes monotone operator theory to conduct analysis for the random updating scheme.…”

Section: B Asynchronous Pdmmmentioning

confidence: 99%

“…Theoretical convergence analyses of PDMM are provided in [13] and [21]. The work of [13] uses the variational inequality to conduct the analysis while [21] makes use of monotone operator theory [22], [23] to interpret PDMM as a Peaceman-Rachford splitting method.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

On Relationship between Primal-Dual Method of Multipliers and Kalman Filter

Zhang,

Kleijn,

Heusdens

2017

Preprint

Self Cite

View full text Add to dashboard Cite

Recently the primal-dual method of multipliers (PDMM), a novel distributed optimization method, was proposed for solving a general class of decomposable convex optimizations over graphic models. In this work, we first study the convergence properties of PDMM for decomposable quadratic optimizations over tree-structured graphs. We show that with proper parameter selection, PDMM converges to its optimal solution in finite number of iterations. We then apply PDMM for the causal estimation problem over a statistical linear state-space model. We show that PDMM and the Kalman filter have the same update expressions, where PDMM can be interpreted as solving a sequence of quadratic optimizations over a growing chain graph.

show abstract

“…Convergence analyses for synchronous PDMM have been provided in [13] and [21]. We present one convergence property from [13] in a theorem below.…”

Section: A Synchronous Pdmmmentioning

confidence: 99%

Section: A Synchronous Pdmmmentioning

confidence: 99%

Section: B Asynchronous Pdmmmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

On Relationship between Primal-Dual Method of Multipliers and Kalman Filter

Zhang,

Kleijn,

Heusdens

2017

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Although ADMM is effective, its convergence rate is often relatively slow because it is based on Douglas-Rachford splitting. The primal-dual method of multipliers (PDMM) [28,29] inherently converges faster as it is based on Peaceman-Rachford splitting [30].…”

mentioning

confidence: 99%

Bregman Monotone Operator Splitting

Niwa,

Kleijn

2018

Preprint

Self Cite

View full text Add to dashboard Cite

Monotone operator splitting is a powerful paradigm that facilitates parallel processing for optimization problems where the cost function can be split into two convex functions. We propose a generalized form of monotone operator splitting based on Bregman divergence. We show that an appropriate design of the Bregman divergence leads to faster convergence than conventional splitting algorithms. The proposed Bregman monotone operator splitting (B-MOS) is applied to an application to illustrate its effectiveness. B-MOS was found to significantly improve the convergence rate.

show abstract

Revisiting the Primal-Dual Method of Multipliers for Optimisation over Centralised Networks

Zhang¹,

Niwa²,

Kleijn³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

The primal-dual method of multipliers (PDMM) was originally designed for solving a decomposable optimisation problem over a general network. In this paper, we revisit PDMM for optimisation over a centralized network. We first note that the recently proposed method FedSplit [1] implements PDMM for a centralized network. In [1], Inexact FedSplit (i.e., gradient based FedSplit) was also studied both empirically and theoretically. We identify the cause for the poor reported performance of Inexact FedSplit, which is due to the improper initialisation in the gradient operations at the client side. To fix the issue of Inexact FedSplit, we propose two versions of Inexact PDMM, which are referred to as gradient-based PDMM (GPDMM) and accelerated GPDMM (AGPDMM), respectively. AGPDMM accelerates GPDMM at the cost of transmitting two times the number of parameters from the server to each client per iteration compared to GPDMM. We provide a new convergence bound for GPDMM for a class of convex optimisation problems. Our new bounds are tighter than those derived for Inexact FedSplit. We also investigate the update expressions of AGPDMM and SCAFFOLD to find their similarities. It is found that when the number K of gradient steps at the client side per iteration is K = 1, both AGPDMM and SCAFFOLD reduce to vanilla gradient descent with proper parameter setup. Experimental results indicate that AGPDMM converges faster than SCAFFOLD when K > 1 while GPDMM converges slightly worse than SCAFFOLD.

show abstract

Derivation and Analysis of the Primal-Dual Method of Multipliers Based on Monotone Operator Theory

Cited by 3 publications

References 0 publications

On Relationship between Primal-Dual Method of Multipliers and Kalman Filter

On Relationship between Primal-Dual Method of Multipliers and Kalman Filter

Bregman Monotone Operator Splitting

Revisiting the Primal-Dual Method of Multipliers for Optimisation over Centralised Networks

Contact Info

Product

Resources

About