Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers

Abstract

Many problems of recent interest in statistics and machine learning can be posed in the framework of convex optimization. Due to the explosion in size and complexity of modern datasets, it is increasingly important to be able to solve problems with a very large number of features or training examples. As a result, both the decentralized collection or storage of these datasets as well as accompanying distributed solution methods are either necessary or at least highly desirable. In this review, we argue that the alternating direction method of multipliers is well suited to distributed convex optimization, and in particular to large-scale problems arising in statistics, machine learning, and related areas. The method was developed in the 1970s, with roots in the 1950s, and is equivalent or closely related to many other algorithms, such as dual decomposition, the method of multipliers, Douglas-Rachford splitting, Spingarn's method of partial inverses, Dykstra's alternating projections, Bregman iterative algorithms for 1 problems, proximal methods, and others. After briefly surveying the theory and history of the algorithm, we discuss applications to a wide variety of statistical and machine learning problems of recent interest, including the lasso, sparse logistic regression, basis pursuit, covariance selection, support vector machines, and many others. We also discuss general distributed optimization, extensions to the nonconvex setting, and efficient implementation, including some details on distributed MPI and Hadoop MapReduce implementations.

Joint relay selection and max‐min energy‐efficient power allocation in downlink multicell NOMA networks: A matching‐theoretic approach

“…However, it would be both interesting and challenging to consider the case of intercell interference due to the frequency reuse in neighboring cells when selecting relays. Another important research direction worth pursuing is the application of solution techniques, as Benders decomposition, 40 the alternating direction method of multipliers, 41 and branch-and-bound, 42 to solve the joint relay selection and max-min energy-efficient power allocation problem. Lastly, communication signaling and overheads between multiple BSs are yet to be fully studied.…”

Integrative Multi-View Regression: Bridging Group-Sparse and Low-Rank Models

“…The convex optimization of (3) has no closed-form solution in general, for which we propose an ADMM algorithm (Boyd et al, 2011). Due to space limit, all the details are presented in Web Appendix A; there we also provide details on handling incomplete data and binary responses as an example of the GLM setup, and on further extensions including the incorporation of 2 regularization and adaptive estimation.…”

Proximal methods for calibration transfer

“…The function L can be minimized using the constrained optimization algorithm alternating direction method of multipliers (ADMM). 12 In a general form, the ADMM algorithm iterates over the 3 steps:…”

“…For extremely large spectra with thousands of channels or more, it may be computationally advantageous to distribute the problem across channels. This is a detailed in Section 8.3 of Boyd et al 12…”