Memory-Efficient Convex Optimization for Self-Dictionary Separable Nonnegative Matrix Factorization: A Frank-Wolfe Approach

Abstract

Nonnegative matrix factorization (NMF) often relies on the separability condition for tractable algorithm design. Separability-based NMF is mainly handled by two types of approaches, namely, greedy pursuit and convex programming. A notable convex NMF formulation is the so-called selfdictionary multiple measurement vectors (SD-MMV), which can work without knowing the matrix rank a priori, and is arguably more resilient to error propagation relative to greedy pursuit. However, convex SD-MMV renders a large memory cost that scales quadratically with the problem size. This memory challenge has been around for a decade, and a major obstacle for applying convex SD-MMV to big data analytics. This work proposes a memory-efficient algorithm for convex SD-MMV. Our algorithm capitalizes on the special update rules of a classic algorithm from the 1950s, namely, the Frank-Wolfe (FW) algorithm. It is shown that, under reasonable conditions, the FW algorithm solves the noisy SD-MMV problem with a memory cost that grows linearly with the amount of data. To handle noisier scenarios, a smoothed group sparsity regularizer is proposed to improve robustness while maintaining the low memory footprint with guarantees. The proposed approach presents the first linear memory complexity algorithmic framework for convex SD-MMV based NMF. The method is tested over a couple of unsupervised learning tasks, i.e., text mining and community detection, to showcase its effectiveness and memory efficiency.

Fast and Structured Block-Term Tensor Decomposition For Hyperspectral Unmixing

“…The identifiability problem has been studied extensively, primarily from a CG-based simplex-structured MF (SSMF) viewpoint [23], [26]. In a nutshell, it has been established that if the abundance matrix S satisfies certain geometric conditions, namely, the pure pixel condition [3], [5], [6], [15], [16] and the sufficiently scattered condition [7], [9], [12], [14], [17], [23], then C and S can be identified up to column and row permutations, respectively. These important results reflect the long postulations in the HU community, i.e., the Winter's and Craig's beliefs [11], [13].…”

“…Under the LMM, the endmembers and abundances can be considered as the two "latent factors" of a matrix factorization (MF) model-which are non-identifiable in general. The remedy is to exploit the convex geometry (CG) of the abundances, e.g., the existence of the so-called "pure pixels" [3], [5], [6], [15], [16], [22] or the sufficient scattering condition [9], [23], [24]. CG-based identfibiaility analysis under the LMM has also influenced many BSS problems in other domains-in particular, machine learning-where similar models are around [23], [25] The CG and MF based HU algorithms have enjoyed many successes [23], [26].…”

Heavy Ball Momentum for Conditional Gradient

“…Throughout, we let x * ∈ X denote a minimizer of (1). FW and its variants are prevalent in various machine learning and signal processing applications, such as traffic assignment [12], non-negative matrix factorization [30], video colocation [17], image reconstruction [15], particle filtering [19], electronic vehicle charging [36], recommender systems [11], optimal transport [26], and neural network pruning [34]. The popularity of FW is partially due to the elimination of projection compared with projected gradient descent (GD) [29], leading to computational efficiency especially when d is large.…”