An alternating direction and projection algorithm for structure-enforced matrix factorization

Abstract

Structure-enforced matrix factorization (SeMF) represents a large class of mathematical models appearing in various forms of principal component analysis, sparse coding, dictionary learning and other machine learning techniques useful in many applications including neuroscience and signal processing. In this paper, we present a unified algorithm framework, based on the classic alternating direction method of multipliers (ADMM), for solving a wide range of SeMF problems whose constraint sets permit low-complexity projections. We propose a strategy to adaptively adjust the penalty parameters which is the key to achieving good performance for ADMM. We conduct extensive numerical experiments to compare the proposed algorithm with a number of state-of-the-art special-purpose algorithms on test problems including dictionary learning for sparse representation and sparse nonnegative matrix factorization. Results show that our unified SeMF algorithm can solve different types of factorization problems as reliably and as efficiently as special-purpose algorithms. In particular, our SeMF algorithm provides the ability to explicitly enforce various combinatorial sparsity patterns that, to our knowledge, has not been considered in existing approaches.

On Indefinite Quadratic Optimization over the Intersection of Balls and Linear Constraints

“…As we see, if ρ ≥ −λ min (A), then in Step 1 and Step 2, we have convex optimization problems, where λ min (A) is the smallest eigenvalue of A. It should also be noted that the convergence results for the ADMM algorithms under some mild assumptions are established in [10,20,30,32] for different classes of optimization problems. The convergence of ADMM to the first-order stationary point is given in the following theorem.…”

“…The BB algorithm of [7] is a recent efficient algorithm to solve p-eTRS that we use in our numerical experiments. Also, we utilize the ADMM approach that has been widely used to solve various classes of optimization problems [3,10,19,20,30,32]. Consider the following ith (m + p − 1)−eTRS (i ∈ I) that arises in the QOBL algorithm:…”

Parallel Algorithms for Constrained Tensor Factorization via Alternating Direction Method of Multipliers

“…ADMoM can easily handle certain structural constraints on the latent factors [26], [33]. For example, if we want to solve an NTF problem with the added constraint that the number of nonzero elements of A is lower than or equal to a given number c A , then we can adopt an approach similar to that followed in Section IV with the only difference being that, instead of using g(Ã) defined in (9), we use g c A (Ã)…”

An efficient algorithm for updating the penalty parameter of the alternating direction method for l1 problem

“…LADM accelerated the algorithm speed by updating the parameter when the relevant error of the current parameter value was not fully improved. In 2017, Xu [20] et al proposed a rule to adjust the adaptive penalty parameters to achieve better performance of ADM for the matrix factorization.…”