Nonlinear Spike-And-Slab Sparse Coding for Interpretable Image Encoding

Abstract

Sparse coding is a popular approach to model natural images but has faced two main challenges: modelling low-level image components (such as edge-like structures and their occlusions) and modelling varying pixel intensities. Traditionally, images are modelled as a sparse linear superposition of dictionary elements, where the probabilistic view of this problem is that the coefficients follow a Laplace or Cauchy prior distribution. We propose a novel model that instead uses a spike-and-slab prior and nonlinear combination of components. With the prior, our model can easily represent exact zeros for e.g. the absence of an image component, such as an edge, and a distribution over non-zero pixel intensities. With the nonlinearity (the nonlinear max combination rule), the idea is to target occlusions; dictionary elements correspond to image components that can occlude each other. There are major consequences of the model assumptions made by both (non)linear approaches, thus the main goal of this paper is to isolate and highlight differences between them. Parameter optimization is analytically and computationally intractable in our model, thus as a main contribution we design an exact Gibbs sampler for efficient inference which we can apply to higher dimensional data using latent variable preselection. Results on natural and artificial occlusion-rich data with controlled forms of sparse structure show that our model can extract a sparse set of edge-like components that closely match the generating process, which we refer to as interpretable components. Furthermore, the sparseness of the solution closely follows the ground-truth number of components/edges in the images. The linear model did not learn such edge-like components with any level of sparsity. This suggests that our model can adaptively well-approximate and characterize the meaningful generation process.

Encoding and Decoding Models in Cognitive Electrophysiology

“…Alternatively, one could create features that exploit the stimulus statistics, for example features that are made statistically independent from each other (Bell and Sejnowski, 1995 ) or by exploiting the concept of sparsity of stimulus representation bases (Olshausen and Field, 1997 , 2004 ; Shelton et al, 2015 ). Feature sparseness of can improve the predictive power and interpretability of models because the representation of stimulus features in active neural populations may be inherently sparse (Olshausen and Field, 2004 ).…”

The Spike-and-Slab Lasso Generalized Linear Models for Prediction and Associated Genes Detection

“…The spike-and-slab prior is the fundamental basis for most Bayesian variable selection approaches, and has proved remarkably successful McCulloch 1993, 1997;Chipman 1996;Chipman et al 2001;Ročková and George 2014, and unpublished results). Recently, Bayesian spike-and-slab priors have been applied to predictive modeling and variable selection in largescale genomic studies (Yi et al 2003;Ishwaran and Rao 2005;de los Campos et al 2010;Zhou et al 2013;Lu et al 2015;Shankar et al 2015;Shelton et al 2015;Partovi Nia and Ghannad-Rezaie 2016). However, most previous spikeand-slab variable selection approaches use the mixture normal priors on coefficients and employ Markov Chain Monte Carlo (MCMC) algorithms (e.g., stochastic search variable selection) to fit the model.…”

GP-Select: Accelerating EM Using Adaptive Subspace Preselection

“…In previous work, the selection function S(y (n) ) was a deterministic function derived individually for each model (see e.g. Shelton et al, 2011Shelton et al, , 2012Dai and Lücke, 2012a,b;Bornschein et al, 2013;Sheikh et al, 2014;Shelton et al, 2015). We now generalize the selection approach: instead of predefining the form of S for variable selection, we want to learn it in a black-box and model-free way based on the data.…”

“…As hyperparameters of kernels are learned, the composition kernel (4) (Shelton et al, 2011(Shelton et al, , 2012(Shelton et al, , 2015. We run all models until convergence.…”