Laplace Group Sensing for Acoustic Models

Abstract

This paper presents the group sparse learning for acoustic models where a sequence of acoustic features is driven by Markov chain and each feature vector is represented by groups of basis vectors. The group of common bases represents the features across Markov states within a regression class. The group of individual basis compensates the intra-state residual information. Laplace distribution is used as the sparse prior of sensing weights for group basis representation. Laplace parameter serves as regularization parameter or automatic relevance determination which controls the selection of relevant bases for acoustic modeling. The groups of regularization parameters and basis vectors are estimated from training data by maximizing the marginal likelihood over sensing weights which is implemented by Laplace approximation using the Hessian matrix and the maximum a posteriori parameters. Model uncertainty is compensated through full Bayesian treatment. The connection of Laplace group sensing to lasso regularization is illustrated. Experiments on noisy speech recognition show the robustness of group sparse acoustic models in presence of different noise types and SNRs.

Neural Bayesian Information Processing

“…The paradigm on combination of traditional and modern machine learning based on probabilistic model and neural network [8] is addressed, respectively. The second section surveys a number of Bayesain methods ranging from latent variable model to variational inference [5,9,36], sampling method [4,6,7], deep unfolding [11] and Bayesian neural network [29]. In the third section, a series of advanced deep models including endto-end memory network [12,37], sequence-to-sequence network [20,22], convolutional network [8,16,25,38], dilated network [2] and attention network [13,17,35] are introduced.…”

Deep Bayesian Mining, Learning and Understanding

“…A new paradigm called the symbolic neural learning is introduced to extend how data analysis is performed from language processing to semantic learning and memory networking. Secondly, we address a number of Bayesian models ranging from latent variable model to VB inference (Chien and Chang, 2014;Chien and Chueh, 2011;Chien, 2015b), MCMC sampling (Watanabe and Chien, 2015) and BNP learning (Chien, 2016;Chien, 2015a;Chien, 2018) for hierarchical, thematic and sparse topics from natural language. In the third part, a series of deep models including deep unfolding (Chien and Lee, 2018), Bayesian RNN (Gal and Ghahramani, 2016;Chien and Ku, 2016), sequence-to-sequence learning (Graves et al, 2006;Gehring et al, 2017), CNN (Kalchbrenner et al, 2014;Xingjian et al, 2015;, GAN (Tsai and Chien, 2017) and VAE are introduced.…”

Balson: Bayesian Least Squares Optimization With Nonnegative L1-Norm Constraint

“…With the estimated posterior distributions, the original parameters can be effectively reconstructed in polynomial fitting problems, and the BALSON framework is found to perform better than conventional methods.Index Terms-Bayesian learning, least squares optimization, L 1 -norm constraint, Dirichlet distribution, sampling method 1. INTRODUCTION In machine learning and statistics, optimization methods, including Newton's method [1], quasi-Newton method [1], sequence quadratic programming (SQP) method [2], gradient descent method [3], interior-point (IP) method [4], and Bayesian methods [5,6,7], are widely applied. The least squares optimization (LSO), which is one of the unconstrained optimization problems, includes the residual sum of squares (RSS) errors as the objective function.…”

“…For example, with the L 1 -norm constraint, the prior distribution is usually assumed to be a Laplacian [6,13,14]. Chien [5] proposed a Bayesian framework based on the Laplace prior of model parameters for sparse representation of sequential data. Finding the mode of the posterior distribution for Gaussian likelihood and Laplacian prior can solve the sparse optimization problem with numerical simulation.There exists another type of regularization with nonnegative L 1 -norm constraint, i.e., the regularization term contains nonnegative elements only [9].…”

“…Finding the mode of the posterior distribution for Gaussian likelihood and Laplacian prior can solve the sparse optimization problem with numerical simulation.There exists another type of regularization with nonnegative L 1 -norm constraint, i.e., the regularization term contains nonnegative elements only [9]. Nonnegative constraint plays an important role for solving the general nonnegative linear or nonlinear programming problems in physics (for example, fluid physics) [15] and engineering applications (for example, hyperspectral image processing, audio processing, web documents analysis, and bioinformatics data processing) [5,10,16]. In this case, Laplacian assumption cannot describe the constraint well as it has negative support.In this paper, we propose a Bayesian learning framework to solve this LSO problem with nonnegative L 1 -norm con-…”