Analysis of the Convergence Speed of the Arimoto-Blahut Algorithm by the Second-Order Recurrence Formula

Nakagawa, Kenji; Takei, Yoshinori; Hara, Shin-ichiro; Watabe, Kohei

doi:10.1109/tit.2021.3095406

Cited by 8 publications

(8 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…By Theorem 4 and 5, the convergence rates are locally linear. Remarkably, the BA-typed algorithm, often serves as a benchmark, is also known to be convergent with linear rate [30]. We therefore demonstrate that the proposed two IB solvers achieve comparable rate of convergence as the BA-typed solver.…”

Section: A Two-block Drs Solvers For Ibmentioning

confidence: 68%

“…)) 2 through the first-order minimizer condition (30) and A being positive definite. This in turn balances the first negative squared norm in the r.h.s.…”

Section: Resultsmentioning

confidence: 99%

“…Then by substituting the first-order necessary conditions for minimizers (30) into the gradient of F, G, along with the relation Ap * = Bq * at a stationary point, we obtain the following relation:…”

Section: Resultsmentioning

confidence: 99%

“…Moreover, our convergence analysis is beyond the IB and PF problems as we relax several assumptions in the recent non-convex DRS convergence analysis [26], [29], [19]. Furthermore, we prove the rate of convergence of the proposed algorithms as locally linear, and hence asymptotically equivalent to that of benchmark solvers [30], through the KŁ inequality [31], [20], [32]. Based on the results, we develop new splitting methods based solvers for IB and PF.…”

Section: Contributionsmentioning

confidence: 86%

“…APPENDIX L PROOF OF LEMMA 14 By the definition (5), the properties of F and G, following algorithm (6) with the first order minimizer conditions (30), and denoting L k c := L c (p k , q k , ν k ) for simplicity, we have:…”

Section: Appendix K Proof Of Lemma 13mentioning

confidence: 99%

See 4 more Smart Citations

A Linearly Convergent Douglas-Rachford Splitting Solver for Markovian Information-Theoretic Optimization Problems

Huang¹,

Gamal²,

Gamal³

2022

Preprint

View full text Add to dashboard Cite

In this work, we propose solving the Information bottleneck (IB) and Privacy Funnel (PF) problems with Douglas-Rachford Splitting methods (DRS). We study a general Markovian information-theoretic Lagrangian that formulates the IB and PF problems into a convex-weakly convex pair of functions in a unified framework. Exploiting the recent non-convex convergence analysis for splitting methods, we prove the linear convergence of the proposed algorithms using the Kurdyka-Łojasiewicz inequality. Moreover, our analysis is beyond IB and PF and applies to any convex-weakly convex pair objectives. Based on the results, we develop two types of IB solvers, with one improves the performance of convergence over existing solvers while the other is linearly convergent independent to the relevance-compression trade-off, and a class of PF solvers that can handle both random and deterministic mappings. Empirically, we evaluate the proposed DRS solvers for both the IB and PF problems with our gradientdescent-based implementation. For IB, the proposed solvers result in solutions that are comparable to those obtained through the Blahut-Arimoto-based benchmark and is convergent for a wider range of the penalty coefficient than existing solvers. For PF, our non-greedy solvers can explore the information plane better than the clustering-based greedy solvers. TABLE I SUMMARY OF THE CONVERGENCE ANALYSIS FOR TWO-BLOCK NON-CONVEX SPLITTING METHODS Reference Algorithm Convergence Conditions Rate of Conv. Properties of Functions Linear Constraints Ap − Bq2α)µ 2 B * locally linear F:convex G:σ G -weakly convex G is Lq-smooth A:full row rank B:positive definite Algorithm (7) 0 < α < 2 c > Mq[ ασ G Mq +φq 4−2α ] † locally linear F :convex G:σ G -weakly convex G:Mq-Lipschitz continuous G are Lq-smooth A:positive definite, B:full row rank Jia et al. [26] Prox. ADMM (α = 1) c > σ G + σ 2 G +8L 2 q 2µ 2 B locally linear F :convex G:σ G -weakly convex G is Lq-smooth A:positive definite B:positive definite Themelis et al. [19] 0 < α < 2; 2 ≤ α < 4 c > Lp; α/Lp−δp 4

show abstract

Section: A Two-block Drs Solvers For Ibmentioning

confidence: 68%

“…)) 2 through the first-order minimizer condition (30) and A being positive definite. This in turn balances the first negative squared norm in the r.h.s.…”

Section: Resultsmentioning

confidence: 99%

Section: Resultsmentioning

confidence: 99%

Section: Contributionsmentioning

confidence: 86%

Section: Appendix K Proof Of Lemma 13mentioning

confidence: 99%

See 3 more Smart Citations

A Linearly Convergent Douglas-Rachford Splitting Solver for Markovian Information-Theoretic Optimization Problems

Huang¹,

Gamal²,

Gamal³

2022

Preprint

View full text Add to dashboard Cite

show abstract

Computation of Rate-Distortion-Perception Function under f-Divergence Perception Constraints

Serra,

Stavrou,

Kountouris

2023

2023 IEEE International Symposium on Information Theory (ISIT)

View full text Add to dashboard Cite

We study the computation of the rate-distortion-perception function (RDPF) for discrete memoryless sources subject to a single-letter average distortion constraint and a perception constraint that belongs to the family of f -divergences. In this setting, the RDPF forms a convex programming problem for which we characterize the optimal parametric solutions. We employ the developed solutions in an alternating minimization scheme, namely Optimal Alternating Minimization (OAM), for which we provide convergence guarantees. Nevertheless, the OAM scheme does not lead to a direct implementation of a generalized Blahut-Arimoto (BA) type of algorithm due to the presence of implicit equations in the structure of the iteration. To overcome this difficulty, we propose two alternative minimization approaches whose applicability depends on the smoothness of the used perception metric: a Newtonbased Alternating Minimization (NAM) scheme, relying on Newton's root-finding method for the approximation of the optimal iteration solution, and a Relaxed Alternating Minimization (RAM) scheme, based on a relaxation of the OAM iterates. Both schemes are shown, via the derivation of necessary and sufficient conditions, to guarantee convergence to a globally optimal solution. We also provide sufficient

show abstract

New Algorithms for Computing Sibson Capacity and Arimoto Capacity

Kamatsuka,

Ishikawa,

Kazama

et al. 2024

2024 IEEE International Symposium on Information Theory (ISIT)

View full text Add to dashboard Cite

Analysis of the Convergence Speed of the Arimoto-Blahut Algorithm by the Second-Order Recurrence Formula

Cited by 8 publications

References 12 publications

A Linearly Convergent Douglas-Rachford Splitting Solver for Markovian Information-Theoretic Optimization Problems

A Linearly Convergent Douglas-Rachford Splitting Solver for Markovian Information-Theoretic Optimization Problems

Computation of Rate-Distortion-Perception Function under f-Divergence Perception Constraints

New Algorithms for Computing Sibson Capacity and Arimoto Capacity

Contact Info

Product

Resources

About