Toward Fast Transform Learning

Abstract

The dictionary learning problem aims at finding a dictionary of atoms that best represents an image according to a given objective. The most usual objective consists of representing an image or a class of images sparsely. Most algorithms performing dictionary learning iteratively estimate the dictionary and a sparse representation of images using this dictionary. Dictionary learning has led to many state of the art algorithms in image processing. However, its numerical complexity restricts its use to atoms with a small support since the computations using the constructed dictionaries require too much resources to be deployed for large scale applications.In order to alleviate these issues, this paper introduces a new strategy to learn dictionaries composed of atoms obtained as a composition of K convolutions with S-sparse kernels. The dictionary update step associated with this strategy is a non-convex optimization problem. We reformulate the problem in order to reduce the number of its irrelevant stationary points and introduce a Gauss-Seidel type algorithm, referred to as Alternative Least Square Algorithm, for its resolution. The search space of the considered optimization problem is of dimension KS, which is typically smaller than the size of the target atom and is much smaller than the size of the image. The complexity of the algorithm is linear with regard to the size of the image.Our experiments show that we are able to approximate with a very high accuracy many atoms such as modified DCT, curvelets, sinc functions or cosines when K is large (say K = 10). We also argue empirically that, maybe surprisingly, the algorithm generally converges to a global minimum for large values of K and S.

Chasing butterflies: In search of efficient dictionaries

“…One drawback with this formulation is that the learned dictionary is highly biased toward the base dictionary, which decreases adaptability to the training data. In [5], the authors propose to learn a dictionary in which each atom is the composition of several circular convolutions using sparse kernels with known supports, so that the dictionary is fast to manipulate. Their problem can be seen as (2), with the gjs corresponding to the M leftmost factors imposing sparse circulant matrices.…”

“…To avoid scaling ambiguities, it is common [5,9] to normalize the factors and introduce a multiplicative scalar λ in the data fidelity term. This results in the optimization problem:…”

“…To handle this cost function, the authors propose an algorithm called Proximal Alternating Linearized Minimization (PALM) [14], that updates alternatively each block of variable by a proximal (or projected in our case) gradient step. PALM can be instantiated to handle the objective (4): there is indeed a match between (4) and (5) …”

“…Generalizing approaches introduced recently in [4] and [5], we propose here a strategy to tackle this problem, through the notion of Flexible Approximate MUlti-layer Sparse Transform (FAµST). We build on the simple observation that the fast transforms associated with analytic dictionaries can be seen as sequences of cheap linear transformations, indicating that such dictionaries can be expressed as 1 [6, Appendix A]:…”

Learning fast sparsifying overcomplete dictionaries

“…For example, square dictionaries (orthonormal and general) that are also known as transforms with complexity O(n log n) have been introduced in [10], [11] and [12]. For more general dictionaries, previous work has explored several ways to introduce structure that improves the computationally complexity of manipulating the dictionary: [13], [14], [15] and [16] to name a few.…”