Distributed strongly convex optimization

Abstract

A lot of effort has been invested into characterizing the convergence rates of gradient based algorithms for non-linear convex optimization. Recently, motivated by large datasets and problems in machine learning, the interest has shifted towards distributed optimization. In this work we present a distributed algorithm for strongly convex constrained optimization. Each node in a network of n computers converges to the optimum of a strongly convex, L-Lipchitz continuous, separable objective at a rate O log ( √ nT ) Twhere T is the number of iterations. This rate is achieved in the online setting where the data is revealed one at a time to the nodes, and in the batch setting where each node has access to its full local dataset from the start. The same convergence rate is achieved in expectation when the subgradients used at each node are corrupted with additive zero-mean noise.

Algorithm-Dependent Generalization Bounds for Multi-Task Learning

“…the strong convexity (Hazan & Kale, 2011) and strong smoothness are dual properties, strongly convex programming algorithms have many benign properties both on the speed of optimization and the quality of generalization; see, for examples, (Hazan & Kale, 2011;Rakhlin et al, 2012;Tsianos & Rabbat, 2012;Kakade & Tewari, 2009). …”

Linearly convergent decentralized consensus optimization with the alternating direction method of multipliers

“…Distributed subgradient descent fits for asynchronous networks, but suffers from slow con vergence. The descent rate of objective value is typically O(log(k)/k) where k is the number of iterations [12]. The ADMM generally needs synchronous steps taken by all the agents, but has much faster empirical convergence.…”

“…Proof Subtracting the three equations in (8) from the cor responding equations in (6) yields \7f(xk +l ) -\7f(x*) = cM + (zk -zk +l ) -M_ ((3k+l -(3* ), (12) � M!. (xk+l -x*) = (3k+l -(3k , (13) �MJ (Xk+l -x*) = Zk +l -z*, (14) respectively.…”

“…(xk+l -x*) = (3k+l -(3k , (13) �MJ (Xk+l -x*) = Zk +l -z*, (14) respectively. Therefore, we can bound I l xk +l -x* II § with mf ll xk +l -x* ll § < (xk+lx*, \7 f (xkH ) -\7 f (x*)) (xk+lx*, cM + (zk -Zk +l ) -M_ ((3k +l -(3* )) (xk+l -x*, cM + (zk -Zk +l )) + (xk+1 -x* , _ M_ ((3k +l -(3* )) 2c(zk -zk +1 ,Zk +l -z*) + % ((3k -(3k+l , (3k+l -(3* ) 2( u ku k+l ) T G ( u k+lu *) I l u ku * llbllu k+1u * llb -I l u ku k+l llb , (15) where the inequality follows from the strong convexity of f(x); the first equality follows from (12); the third equality comes from (13) and (14).…”

“…\7 f(x); the second inequality follows from(12) and the basic inequality I i a + b ll § :::: : ( 1 -J-l ) lla ll� + ( 1 -� ) llbll� , \:I J-l > 0 ; the third inequality holds since we choose (30 in the column space of M!. such that (3k+l and (3* also lie in the column space of M!.…”