Randomized Algorithms for Distributed Nonlinear Optimization Under Sparsity Constraints

Abstract

Distributed optimization in multi-agent systems under sparsity constraints has recently received a lot of attention. In this paper, we consider the in-network minimization of a continuously differentiable nonlinear function which is a combination of local agent objective functions subject to sparsity constraints on the variables. A crucial issue of in-network optimization is the handling of the communications, which may be expensive. This calls for efficient algorithms, that are able to reduce the number of required communication links and transmitted messages. To this end, we focus on asynchronous and randomized distributed techniques. Based on consensus techniques and iterative hard thresholding methods, we propose three methods that attempt to minimize the given function, promoting sparsity of the solution: asynchronous hard thresholding (AHT), broadcast hard thresholding (BHT), and gossip hard thresholding (GHT). Although similar in many aspects, it is difficult to obtain a unified analysis for the proposed algorithms. Specifically, we theoretically prove the convergence and characterize the limit points of AHT in regular networks under some proper assumptions on the functions to be minimized. For BHT and GHT, instead, we characterize the fixed points of the maps that rule their dynamics in terms of stationary points of original problem. Finally, we illustrate the implementation of our techniques in compressed sensing and present several numerical results on performance and number of transmissions required for convergence.

Improved iterative shrinkage-thresholding for sparse signal recovery via Laplace mixtures models

“…The proof can be obtained with similar techniques, devised in [ 51 ], and we omit the proof for brevity. This result provides a necessary condition for optimality and shows that, being the function in ( 12 ) not convex, τ -stationarity points are only local minima.…”

Decentralized Frank–Wolfe Algorithm for Convex and Nonconvex Problems

“…Despite the high communication cost, the per-iteration computation complexity of (39) is also high, i.e., at O(d log d) [41]. We notice that [7,8] have considered distributed sparse recovery algorithm with focus on the communication efficiency. However, their algorithms are based on the iterative hard thresholding (IHT) formulation [52] that requires a-priori knowledge on the sparsity level of θ true .…”

“…The centralized FW algorithm for both losses will also be compared (cf. (7)); as well as the decentralized algorithm in [45] (labeled as 'Qing et al') and the DPG algorithm [15] with step size set to α t = 0.1N/( √ t + 1) applied to square loss. Our first example considers the noiseless synthetic dataset of problem dimension m 1 = 100, m 2 = 250, K = 5.…”

“…The sparsified DeFW algorithm is implemented with p t = 2 + α comm · t , t = log(t) + 1 with extreme or random coordinate selection. We compare the algorithms of PG-EXTRA [16] (with fixed step size α = 1/d), DPG [15] (with step size α t = 1/t) and BHT [7]. DeFW, PG-EXTRA and DPG are set to solve the convex problem (38) with R = 1.1 θ true 1 .…”

“…Typically, the function f i (θ) models the loss function over the private data held by agent i, and C corresponds to a regularization constraint imposed on the optimization variable θ such as sparsity or low rank of an array of values. Problem (1) covers a number of applications in control theory, signal processing and machine learning, including system identification [5], matrix completion [6] and sparse learning [7,8].…”