Trainlets: Dictionary Learning in High Dimensions

Abstract

Sparse representations has shown to be a very powerful model for real world signals, and has enabled the development of applications with notable performance. Combined with the ability to learn a dictionary from signal examples, sparsity-inspired algorithms are often achieving state-of-the-art results in a wide variety of tasks. Yet, these methods have traditionally been restricted to small dimensions mainly due to the computational constraints that the dictionary learning problem entails. In the context of image processing, this implies handling small image patches. In this work we show how to efficiently handle bigger dimensions and go beyond the small patches in sparsity-based signal and image processing methods. We build our approach based on a new cropped wavelet decomposition, which enables a multi-scale analysis with virtually no border effects. We then employ this as the base dictionary within a double sparsity model to enable the training of adaptive dictionaries. To cope with the increase of training data, while at the same time improving the training performance, we present an Online Sparse Dictionary Learning (OSDL) algorithm to train this model effectively, enabling it to handle millions of examples. This work shows that dictionary learning can be up-scaled to tackle a new level of signal dimensions, obtaining large adaptable atoms that we call trainlets

Orthogonal Procrustes Analysis for Dictionary Learning in Sparse Linear Representation

“…Here we extend the comparative experiments considering R-SVD (integrating OMP) versus the iterative alternating scheme method ILS-DLA, and the on-line dictionary learning method OSDL by Sulam et al [24]. …”

“…Then we apply the method on synthetic data conducting extensive experiments on both R-SVD and K-SVD using OMP as sparsifier. A further investigation is conducted on two different sparse decomposition methods, namely k -Limaps and SL0, and alternative dictionary learning methods, namely ILS-DLA by Engan et al [20] and OSDL by Sulam et al [24]. …”

“…This is accomplished through rank-1 singular value decomposition of the residual submatrix, accounting for all examples using the atom under consideration. Recently, Sulam et al [24] introduced OSDL, an hybrid version of dictionary design, which builds dictionaries, fast to apply, by imposing a structure based on a multiplication of two matrices, one of which is fully-separable cropped Wavelets and the other is sparse, bringing to a double-sparsity format.…”

Optimization of light fields in ghost imaging using dictionary learning

“…Thus the patch size used in dictionary learning should not be large, which, however, poses a limitation to the image size that we can handle. Nevertheless, efficient dictionary learning methods dealing with images of larger scales have recently been proposed [46][47][48], in which the handled image size can go beyond 64 × 64 pixels. To demonstrate the effectiveness of the proposed method for images of larger size, we carry out simulations over the LFWcrop database [49], which consists of more than 13, 000 images of 64 × 64 pixels.…”

Dictionary learning from incomplete data for efficient image restoration

“…In addition, due to the computational complexity of the problem, the majority of the learning algorithms have been restricted to work with relatively small signals. Recent works [4] tried to address the latter issue by observing that learned dictionaries have a certain sparse structure, which can be efficiently utilised. These novel algorithms are capable of handling signals at the dimension of several thousands, and learn on millions of data signals.…”