On Robustness of the Normalized Subgradient Method with Randomly Corrupted Subgradients

Turan, Berkay; Uribe, César A.; Wai, Hoi-To; Alizadeh, Mahnoosh

doi:10.48550/arxiv.2009.13725

Cited by 2 publications

(4 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We allow arbitrary adversarial corruption in a centralized setup, which prevents robust aggregation to create gradient estimates. Closest to our setup is our previous work in [37], which studies robustness of normalized subgradient method in a randomly corrupted subgradient setting. However, [37] studies a full gradient type method for constrained convex optimization problems satisfying a certain acute angle condition, whereas this work considers a block coordinate descent type method for unconstrained non-convex optimization problems.…”

Section: Introductionmentioning

confidence: 99%

“…Closest to our setup is our previous work in [37], which studies robustness of normalized subgradient method in a randomly corrupted subgradient setting. However, [37] studies a full gradient type method for constrained convex optimization problems satisfying a certain acute angle condition, whereas this work considers a block coordinate descent type method for unconstrained non-convex optimization problems. Paper Organization: The remainder of the paper is organized as follows.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

On Robustness of the Normalized Random Block Coordinate Method for Non-Convex Optimization

Turan

Uribe

Wai

et al. 2021

2021 60th IEEE Conference on Decision and Control (CDC)

View full text Add to dashboard Cite

Large-scale optimization problems are usually characterized not only by large amounts of data points but points living in a high-dimensional space. Block coordinate methods allow for efficient implementations where steps can be made (block) coordinate-wise. Many existing algorithms rely on trustworthy gradient information and may fail to converge when such information becomes corrupted by possibly adversarial agents. We study the setting where the partial gradient with respect to each coordinate block is arbitrarily corrupted with some probability. We analyze the robustness properties of the normalized random block coordinate method (NRBCM) for non-convex optimization problems. We prove that NRBCM finds an O(1/ √ T )-stationary point after T iterations if the corruption probabilities of partial gradients with respect to each block are below 1/2. With the additional assumption of gradient domination, faster rates are shown. Numerical evidence on a logistic classification problem supports our results.

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

On Robustness of the Normalized Random Block Coordinate Method for Non-Convex Optimization

Turan

Uribe

Wai

et al. 2021

2021 60th IEEE Conference on Decision and Control (CDC)

View full text Add to dashboard Cite

show abstract

“…Seeing the need for large batch sizes for variance reduction of stochastic gradients as a drawback of normalized updates, a recent work [33] proves that adding momentum removes the need for large batch sizes on non-convex objectives while matching the best-known convergence rates. In a preliminary conference report [34], we investigated the robustness properties of the normalized subgradient method for solving deterministic optimization problems in a centralized fashion. In the current work, we expand [34] into a distributed setup with a stochastic objective function, additionally study non-convex objectives both theoretically and numerically, and employ two additional layers of defense by means of robust mean estimation before applying normalization to improve our algorithm.…”

mentioning

confidence: 99%

“…In a preliminary conference report [34], we investigated the robustness properties of the normalized subgradient method for solving deterministic optimization problems in a centralized fashion. In the current work, we expand [34] into a distributed setup with a stochastic objective function, additionally study non-convex objectives both theoretically and numerically, and employ two additional layers of defense by means of robust mean estimation before applying normalization to improve our algorithm.…”

mentioning

confidence: 99%

Robust Distributed Optimization With Randomly Corrupted Gradients

Turan,

Uribe,

Wai

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

In this paper, we propose a first-order distributed optimization algorithm that is provably robust to Byzantine failures-arbitrary and potentially adversarial behavior, where all the participating agents are prone to failure. We model each agent's state over time as a two state Markov chain that indicates Byzantine or trustworthy behaviours at different time instants. We set no restrictions on the maximum number of Byzantine agents at any given time. We design our method based on three layers of defense: 1) Temporal gradient averaging, 2) robust aggregation, and 3) gradient normalization. We study two settings for stochastic optimization, namely Sample Average Approximation and Stochastic Approximation, and prove that for strongly convex and smooth non-convex cost functions, our algorithm achieves order-optimal statistical error and convergence rates. I. INTRODUCTIONConvenience for large-scale data processing, privacy preservation, and parallel algorithm execution rendered the design of distributed optimization algorithms an attractive field for scholars [1]- [7]. However, the distributed nature of such methods, for example, physically separated servers connected over a network, exposes the system to vulnerabilities not faced by their traditional centralized counterparts [8]. The robustness and security of distributed methods need to be taken into account when assessing algorithm performance [2].In a centralized system, data can be cleaned, faultless computation can be established by reliable hardware, and communication requirements are minimal. On the other hand, typical distributed algorithms assume trustworthy data, faultless computation, and reliable communication. Also, privacy constraints might not allow for data corruption checks, while distributed computing infrastructure increases the likelihood of faulty hardware, e.g., personal devices [9]. Lastly, unreliable communication might occur due to noisy wireless communication, or more importantly, due to man-in-the-middle adversarial attacks. In man-in-the-middle attacks, an adversary can take over network sub-systems and arbitrarily alter the information stored in and communicated between the machines to prevent convergence to the optimal solution, i.e., Byzantine attacks [10].Robust distributed optimization under adversarial manipulation has been studied for various corruption models, see [11],

show abstract

On Robustness of the Normalized Subgradient Method with Randomly Corrupted Subgradients

Cited by 2 publications

References 22 publications

On Robustness of the Normalized Random Block Coordinate Method for Non-Convex Optimization

On Robustness of the Normalized Random Block Coordinate Method for Non-Convex Optimization

Robust Distributed Optimization With Randomly Corrupted Gradients

Contact Info

Product

Resources

About