Distributed Newton Method for Large-Scale Consensus Optimization

Abstract

In this paper, we propose a distributed Newton method for consensus optimization. Our approach outperforms state-of-the-art methods, including ADMM. The key idea is to exploit the sparsity of the dual Hessian and recast the computation of the Newton step as one of efficiently solving symmetric diagonally dominant linear equations. We validate our algorithm both theoretically and empirically. On the theory side, we demonstrate that our algorithm exhibits superlinear convergence within a neighborhood of optimality. Empirically, we show the superiority of this new method on a variety of machine learning problems. The proposed approach is scalable to very large problems and has a low communication overhead.Generally, there are two popular classes of algorithms for distributed optimization. The first is sub-gradient based, while the second relies on a decomposition-coordination procedure. Sub-gradient algorithms proceed by taking a gradient related step then followed by an averaging with neighbors at each iteration. The computation of each step is relatively cheap and can be implemented in a distributed fashion [1]. Though cheap to compute, the best known convergence rate of sub-gradient methods is relatively slow given by O 1 √ t with t being the total number of iterations [2,18]. The second class of algorithms solve constrained problems by relying on dual methods. One of the well-know methods (state-of-the-art) from this class is the Alternating Direction Method of Multipliers (ADMM) [3]. ADMM decomposes the original problem to two subproblems which are then solved sequentially leading to updates of dual variables. In [2], the authors show that ADMM can be fully distributed over a network leading to improved convergence rates in the order of O 1 t . Apart from accuracy problems inherent to ADMM-based methods [4], much rate improvements can be gained from adopting second-order (Newton) methods. Though a variety of techniques have been proposed [7,6,5], less progress has been made at leveraging ADMM's accuracy and convergence rate issues. In a recent attempt [9,10], the authors propose a distributed second-order method for general consensus by using the approach in [8] to compute the Newton direction. As detailed in Section 6, this method suffers from two problems. First, it fails to outperform ADMM and second, faces storage and computational defficiencies for large data sets, thus ADMM retains state-of-the-art status 1 . Contributions:In this paper, we contribute to the above problems and propose a distributed Newton method for general consensus with the following characteristics: i) approximating the exact newton direction up-to any arbitrary > 0, ii) exhibiting super-linear convergence within a neighborhood of the optimal solution similar to exact Newton, and iii) outperforming ADMM and others in terms of iteration count, running times, and total message complexity on a set of benchmark datasets, including one on a real-world application of fMRI imaging. One can argue that our improvements arrive at increased commu...

A Chebyshev-Accelerated Primal-Dual Method for Distributed Optimization

“…then (15) and in turn (13) are satisfied, which by Theorem 1.1 of [10] implies (14). Recalling that L F = L f + ρλ n , a simple rearrangement of (16) gives (10).…”

A General Framework of Exact Primal-Dual First-Order Algorithms for Distributed Optimization

“…., x n ] ∈ R n , and Ax = 0 represents all equality constraints. Some choices for matrix A include edge-node incidence matrix [4], weighted incidence matrix [45], graph Laplacian matrix [41], and weighted Laplacian matrix [23,1]. In this paper, we choose matrix A to be the edge-node incidence matrix of the network graph, i.e., A ∈ R ×n , = |E|, whose null space is spanned by the vector of all ones.…”

“…The authors in [9] proposed a distributed primal BFGS algorithm which converges to the exact solution with a linear rate under strong convexity assumption. The authors in [41] proposed a primal-dual algorithm, which minimizes the augmented Lagrangian in the primal space and uses approximate Newton step to update the dual variable. The iterates of this algorithm go through a quadratic phase of convergence and converge to the exact solution.…”

Achieving Acceleration in Distributed Optimization via Direct Discretization of the Heavy-Ball ODE

“…. ; x n ) = n i=1 f i (x i ), and define matrix L ∈ R n×n as the Laplacian matrix of the graph G. It can be easily verified (see [17]) that the constraint x 1 = · · · = x n is equivalent to Lx = 0 where L = L ⊗ I p ∈ R np × R np is the Kronecker product of the Laplacian matrix L and the identity matrix I p . By incorporating these definitions Problem (2) can be written as…”

“…This paper is organized as follows. Section II recalls the problem of distributed optimization over networks and presents its formulation in terms of a set of equality constraints related to the Laplacian of the network as in [17]. Section III describes the dual formulation of the distributed optimization problem and some basic properties of the dual problem.…”