An $\alpha$ -Divergence-Based Approach for Robust Dictionary Learning

Abstract

In this paper, a robust sequential dictionary learning (DL) algorithm is presented. The proposed algorithm is motivated from the maximum likelihood perspective on dictionary learning and its link to the minimization of the Kullback-Leibler divergence. It is obtained by using a robust loss function in the data fidelity term of the DL objective instead of the usual quadratic loss. The proposed robust loss function is derived from the αdivergence as an alternative to the Kullback-Leibler divergence, which leads to a quadratic loss. Compared to other robust approaches, the proposed loss has the advantage of belonging to class of redescending M-estimators, guaranteeing inference stability from large deviations from the Gaussian nominal noise model. The algorithm is obtained by solving a sequence of penalized rank-1 matrix approximation problems where the 1−norm is introduced as a penalty promoting sparsity and then using a block coordinate descent approach to estimate the unknowns. Performance comparison with similar robust DL algorithms on digit recognition, background removal, and grayscale image denoising are performed highlighting efficacy of the proposed algorithm.

Adversarial dictionary learning for a robust analysis and modelling of spontaneous neuronal activity

“…In image and video processing, where it is common to learn dictionaries adapted to small patches, with training data that may include several millions of these patches, Mairal et al [19] proposed an online dictionary learning (ODL) algorithm based on stochastic approximations, which scales up to large datasets with millions of training samples handling also dynamic training data changing over time, such as video sequences. In the same context of image processing, Iqbal et al [23] proposed a DL algorithm, which minimizes the assumption on the noise by using a function derived from the α-divergence, which is used in the data fidelity term of the objective function instead of the quadratic loss or the Frobenius norm. The algorithm is applied on various image processing applications, such as digit recognition, background removal, and grayscale image denoising.…”

The α-divergences associated with a pair of strictly comparable quasi-arithmetic means

“…The α-divergences are widely used in information sciences, see [3,8,38,22,17,1] just to cite a few applications. The singly-parametric α-divergences have also been generalized to bi-parametric families of divergences like the (α, β)-divergences [2] or the αβ-divergences [37].…”

Blind source separation of functional dynamic MRI signals via dictionary learning

“…Concerning the fMRI framework, besides K-SVD, the most commonly used DL algorithm for analyzing fMRI data is the Online Dictionary Learning (ODL) algorithm [96]. Despite its relative simplicity, ODL constitutes a very efficient algorithm that has been successfully applied to reconstruct current brain networks from both task-related and rs-fMRI [1,82,127,75,104]. Furthermore, ODL often acts as a reference baseline for the development of novel advanced DL algorithms [155].…”

“…Therefore, it is not surprising to find out that, during the last years, many different DL algorithms and methods have been developed oriented to analyze fMRI data. For example, there exist DL methods particularly tailored to the fMRI data, such as [82], or another more recent and sophisticated approach adapted to the natural noise present in the fMRI data, such as [75].…”

“…Typically, we can distinguish two approaches based on how they update the dictionary: (a) the whole dictionary matrix is updated at once, i.e., all the atoms of the dictionary are simultaneously updated, for example, using least-squares [52] or the maximum likelihood [84], and (b) each atom of the dictionary is sequentially updated, i.e., the second approach decomposes the dictionary update into K sequential minimization problems, one per atom, which are solved one at a time [75,127,6].…”