Randomized Block Frank–Wolfe for Convergent Large-Scale Learning

Abstract

Abstract-Owing to their low-complexity iterations, FrankWolfe (FW) solvers are well suited for various large-scale learning tasks. When block-separable constraints are present, randomized block FW (RB-FW) has been shown to further reduce complexity by updating only a fraction of coordinate blocks per iteration. To circumvent the limitations of existing methods, the present work develops step sizes for RB-FW that enable a flexible selection of the number of blocks to update per iteration while ensuring convergence and feasibility of the iterates. To this end, convergence rates of RB-FW are established through computational bounds on a primal sub-optimality measure and on the duality gap. The novel bounds extend the existing convergence analysis, which only applies to a step-size sequence that does not generally lead to feasible iterates. Furthermore, two classes of step-size sequences that guarantee feasibility of the iterates are also proposed to enhance flexibility in choosing decay rates. The novel convergence results are markedly broadened to encompass also nonconvex objectives, and further assert that RB-FW with exact line-search reaches a stationary point at rate O(1/ √ t). Performance of RB-FW with different step sizes and number of blocks is demonstrated in two applications, namely charging of electrical vehicles and structural support vector machines. Extensive simulated tests demonstrate the performance improvement of RB-FW relative to existing randomized single-block FW methods.

Nonlinear Dimensionality Reduction for Discriminative Analytics of Multiple Datasets

“…Remark 5. When the kernel function required to form K xx , K yy , and K xy is not given, one may use the multi-kernel learning method to automatically choose the right kernel function(s); see for example, [4], [38], [39]. Specifically, one can presume K xx := P i=1 δ i K i xx , K yy := P i=1 δ i K i yy , and K xy := P i=1 δ i K i xy in (18), where K i xx ∈ R m×m , K i yy ∈ R n×n , and K i xy ∈ R m×n are formed using the kernel function κ i (•); and {κ i (•)} P i=1 are a preselected dictionary of known kernels, but {δ i } P i=1 will be treated as unknowns to be learned along with A in (18).…”

Kernel-Based Inference of Functions Over Graphs

“…. , θ (η) (M )] T can be found by jointly minimizing (48) with Frank-Wolfe algorithm [65]. When the kernel matrices belong to the Laplacian family (16), efficient algorithms that exploit the common eigenspace of the kernels in the dictionary have been developed in [61].…”

A Momentum-Guided Frank-Wolfe Algorithm

“…Providing the easiness in implementation and enabling structural solutions, FW is of interest in various applications. Besides those mentioned earlier, other examples encompass structural SVM [10], video colocation [11], particle filtering [12], traffic assignment [13], and optimal transport [14], electronic vehicle charging [15], [16], and submodular optimization [17].…”