Boltzmann Machine and Mean-Field Approximation for Structured Sparse Decompositions

Abstract

To cite this version:Angélique Drémeau, Cédric Herzet, Laurent Daudet. Boltzmann machine and mean-field approximation for structured sparse decompositions. Accepté à IEEE Trans. On Signal Processing. 2012. <hal-00648089v2> AbstractTaking advantage of the structures inherent in many sparse decompositions constitutes a promising research axis. In this paper, we address this problem from a Bayesian point of view. We exploit a Boltzmann machine, allowing to take a large variety of structures into account, and focus on the resolution of a marginalized maximum a posteriori problem. To solve this problem, we resort to a mean-field approximation and the "variational Bayes Expectation-Maximization" algorithm. This approach results in a soft procedure making no hard decision on the support or the values of the sparse representation. We show that this characteristic leads to an improvement of the performance over state-of-the-art algorithms. Index TermsStructured sparse representation, Bernoulli-Gaussian model, Boltzmann machine, mean-field approximation.

A general framework for dictionary based audio fingerprinting

“…This model recently appeared in dictionary based processing setups. Dremeau et al [15] show that it generalizes many structured sparsity models. Under this model, we can evaluate the probability of a state using the difference of energy for atom i:…”

“…Future work will investigate smarter optimization scheme, such as Bayesian versions of MP (e.g. as in [15] with Boltzmann machines) or convex relaxations methods.…”

Mean-field inference methods for neural networks

“…Subsequently, in the reconstruction of a new signal of the same type, the generative model can serve as a Bayesian prior. In particular, the idea to exploit RBMs in CS applications was pioneered by [DHD12] and [TDK16], who trained binary RBMs using Contrastive Divergence (to locate the support of the non-zeros entries of sparse signals) and combined it with an AMP reconstruction. They demonstrated drastic improvements in the reconstruction with structured learned priors compared to the usual sparse unstructured priors.…”

A greedy algorithm with learned statistics for sparse signal reconstruction

“…• The atom selection step does not take into account the new update step (13). This means that there is no guarantee that the selected atom will lead to a steepest descent of the residual error, as for the OMP.…”

“…"Spike-andslabs" prior models have been introduced [9][10][11], but usually assume that the coefficients are independent and identically distributed, so they do not make use of the statistical dependencies between coefficients. Graphical models have been proposed to model theses dependencies [12][13][14]. However this approach involves the design of specialised, and often computationally expensive optimization methods.…”