Learning Sparsely Used Overcomplete Dictionaries via Alternating Minimization

Abstract

We consider the problem of sparse coding, where each sample consists of a sparse linear combination of a set of dictionary atoms, and the task is to learn both the dictionary elements and the mixing coefficients. Alternating minimization is a popular heuristic for sparse coding, where the dictionary and the coefficients are estimated in alternate steps, keeping the other fixed. Typically, the coefficients are estimated via ℓ 1 minimization, keeping the dictionary fixed, and the dictionary is estimated through least squares, keeping the coefficients fixed. In this paper, we establish local linear convergence for this variant of alternating minimization and establish that the basin of attraction for the global optimum (corresponding to the true dictionary and the coefficients) is O 1/s 2 , where s is the sparsity level in each sample and the dictionary satisfies RIP. Combined with the recent results of approximate dictionary estimation, this yields provable guarantees for exact recovery of both the dictionary elements and the coefficients, when the dictionary elements are incoherent.The problem of sparse coding consists of unsupervised learning of the dictionary and the coefficient matrices. Thus, given only unlabeled data, we aim to learn the set of dictionary atoms or basis functions that provide a good fit to the observed data. Sparse coding is applied in a variety of domains. Sparse coding of natural images has yielded dictionary atoms which resemble the receptive fields of neurons in the visual cortex [26,27], and has also yielded localized dictionary elements on speech and video data [19,25].An important strength of sparse coding is that it can incorporate overcomplete dictionaries, where the number of dictionary atoms r can exceed the observed dimensionality d. It has been argued that having overcomplete representation provides greater flexibility is modeling and more robustness to noise [19], which is crucial for encoding complex signals present in images, speech and video. It has been shown that the performance of most machine learning methods employed downstream is critically dependent on the choice of data representations, and overcomplete representations are the key to obtaining state-of-art prediction results [6].On the downside, the problem of learning sparse codes is computationally challenging, and is in general, NP-hard [9]. In practice, heuristics are employed based on alternating minimization. At a high level, this consists of alternating steps, where the dictionary is kept fixed and the coefficients are updated and vice versa. Such alternating minimization methods have enjoyed empirical success in a number of settings [18,10,2,20,35]. In this paper, we carry out a theoretical analysis of the alternating minimization procedure for sparse coding.

Rounding sum-of-squares relaxations

“…The question of when a nonnegative polynomial has such a "certificate of nonnegativity" was studied by Hilbert, who realized this doesn't always hold and asked (as his 17th problem) whether a nonnegative polynomial is always a sum of squares of rational functions. 1 The book chapter [20] is a good source for several of the known upper and lower bounds, although it does not contain some of the more recent ones. 2 While it is common in the TCS community to use Lasserre to describe the primal version of this SDP and Sum-ofSquares (SOS) to describe the dual, in this paper we use the more descriptive SOS name for both programs.…”

“…There are several strong lower bounds (also known as integrality gaps) for these hierarchies, in particular showing that ω(1) levels (and often even n Ω (1) or Ω(n) levels) of many such hierarchies can't improve by much on the known polynomial-time approximation guarantees for many NP hard problems, including SAT, Independent-Set, Max-Cut and more [28,27,5,21,47,52,19,14,15]. Unfortunately, there are many fewer positive results, and many of them only show that these hierarchies can match the performance of Proceedings of the 2014 ACM Symposium on Theory of Computing 31 Proceedings of the 2014 ACM Symposium on Theory of Computing previously known (and often more efficient) methods, or give algorithms that can be converted into something much more combinatorial, rather than using hierarchies to get genuinely new algorithmic results.…”

“…Unfortunately, there are many fewer positive results, and many of them only show that these hierarchies can match the performance of Proceedings of the 2014 ACM Symposium on Theory of Computing 31 Proceedings of the 2014 ACM Symposium on Theory of Computing previously known (and often more efficient) methods, or give algorithms that can be converted into something much more combinatorial, rather than using hierarchies to get genuinely new algorithmic results. 1 One of the reasons for this paucity of positive results is that we have relatively few tools to round such convex hierarchies. A rounding algorithm maps a solution to the relaxation to a solution to the original program.…”

Generalized Low Rank Models

“…Recently, there has been a resurgence of interest in methods based on alternating minimization, as numerous authors have shown that alternating minimization (suitably initialized, and under a few technical assumptions) provably converges to the global minimum for a range of problems including matrix completion [Kes12,JNS13,Har13], robust PCA [NNS + 14], and dictionary learning [AAJN13].…”

Learning Sparse Representations in Reinforcement Learning with Sparse Coding

“…We then show that, despite being a nonconvex objective, all local minima are global minima, under minimal conditions. We avoid the need for careful initialization strategies needed for previous optimality results for sparse coding [Agarwal et al, 2014;Arora et al, 2015], using recent results for more general dictionary learning settings [Haeffele and Vidal, 2015;Le and White, 2017], particularly by extending beyond smooth regularizers using Γ-convergence. Using this insight, we provide a simple alternating proximal gradient algorithm and demonstrate the utility of learning supervised sparse coding representations versus unsupervised sparse coding and a variety of tile-coding representations.…”