Online Sparsifying Transform Learning—Part II: Convergence Analysis

Abstract

Sparsity-based techniques have been widely popular in signal processing applications such as compression, denoising, and compressed sensing. Recently, the learning of sparsifying transforms for data has received interest. The advantage of the transform model is that it enables cheap and exact computations. In Part I of this work, efficient methods for online learning of square sparsifying transforms were introduced and investigated (by numerical experiments). The online schemes process signals sequentially, and can be especially useful when dealing with big data, and for real-time, or limited latency signal processing applications. In this paper, we prove that although the associated optimization problems are non-convex, the online transform learning algorithms are guaranteed to converge to the set of stationary points of the learning problem. The guarantee relies on a few simple assumptions. In practice, the algorithms work well, as demonstrated by examples of applications to representing and denoising signals.

Online Sparsifying Transform Learning—Part I: Algorithms

“…In practice, depends on , and algorithm initialization, and is typically larger for bigger, or more complex problems. On the other hand, as shown in Part II [45], and in the experiments of Section IV, the online and mini-batch schemes produce good transforms for (total number of signals processed sequentially) large. Therefore, the net computational cost for processing signals (and converging) for the online scheme is .…”

“…As we show in this work, online transform learning involves cheap computations and modest memory requirements. Moreover, in Part II of this work [45], convergence guarantees are provided for online transform learning. Our numerical experiments illustrate the usefulness of our schemes for big data processing (online sparse representation and denoising).…”

“…Furthermore, the proof by Mairal et al [47] of the convergence of online synthesis dictionary learning requires various restrictive assumptions. In contrast, the analysis in Part II (see [45] for details) relies on only a few simple assumptions. Another feature distinguishing our formulation is that in the previous work [47], the objective is biconvex, so that the non-convexity in the problem vanishes when a particular variable is kept fixed.…”

“…In the formula of , the for smaller values gets scaled by smaller numbers (i.e., ). It follows from results in Part II [45] (based on the bound for the rightmost term in (24) of [45]) that decays (in norm) as , for some constant . Based on that result, it is easy to show that the approximation error is bounded by for all .…”

“…In Part II [45], it is shown that online transform learning converges asymptotically, and produces a good transform. Here, we present numerical results illustrating the practical convergence behavior of online (and mini-batch) transform learning, as well as the usefulness of the proposed schemes for image representation and denoising.…”