Hybrid Random/Deterministic Parallel Algorithms for Convex and Nonconvex Big Data Optimization

Abstract

We propose a decomposition framework for the parallel optimization of the sum of a differentiable (possibly nonconvex) function and a nonsmooth (possibly nonseparable), convex one. The latter term is usually employed to enforce structure in the solution, typically sparsity. The main contribution of this work is a novel parallel, hybrid random/deterministic decomposition scheme wherein, at each iteration, a subset of (block) variables is updated at the same time by minimizing a convex surrogate of the original nonconvex function. To tackle huge-scale problems, the (block) variables to be updated are chosen according to a mixed random and deterministic procedure, which captures the advantages of both pure deterministic and random update-based schemes. Almost sure convergence of the proposed scheme is established. Numerical results show that on huge-scale problems the proposed hybrid random/deterministic algorithm compares favorably to random and deterministic schemes on both convex and nonconvex problems.

A survey of machine learning for big data processing

“…For example, in [113], two distributed learning algorithms for training random vector functional-link (RVFL) networks through interconnected nodes were presented, where training data were distributed under a decentralized information structure. To tackle the huge-scale convex and nonconvex big data optimization problems, a novel parallel, hybrid random/deterministic decomposition scheme with the power of dictionary learning was investigated in [114]. In [87], the authors developed a low-complexity, real-time online algorithm for decomposing low-rank tensors with missing entries to deal with the incomplete streaming data, and the performance of the proposed subspace learning was also validated.…”

Inexact Block Coordinate Descent Algorithms for Nonsmooth Nonconvex Optimization

“…We consider two commonly used rules to select the block variable, namely, the cyclic update rule and the random update rule. Note that both of them are well-known (see [18,19]), but we give their definitions for the sake of reference in later developments.…”

“…and k p t k = 1. Any block variable can be selected with a nonzero probability, and some examples are given in [19].…”

“…2. As most iterative algorithms exhibit asymptotic convergence only, in practice, they are terminated when we obtain a point, denoted as x t k , that is "sufficiently accurate" in the sense that the so-called error bound x t k − B k x t is smaller than some given t that decreases to zero as t increases [15,19].…”

“…Several variants are considered, with different number of block variables (K = 1, 2, 10) and inner layer iterations (τ t = 1, 10); • BSCA algorithm with quadratic approximation (cf. (19)), referred to as "BGD" (block gradient descent). Several variants with different number of block variables (K = 2, 10) are considered.…”