A Kernel-Expanded Stochastic Neural Network

Sun, Yan; Liang, Faming

doi:10.1111/rssb.12496

Cited by 3 publications

(2 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The potential of stochastic activation layers for UQ was briefly discussed in [69], but the context of this work was classification tasks, where the predictive uncertainty could already be fully characterized by well-calibrated categorical distributions without using any latent noise. More recently, [70] formulated DNNs as latent variable models and included kernel maps in the input layer to avoid feature collinearity. Although the proposed model was capable of UQ, the more challenging problem of DR was not studied.…”

Section: Related Workmentioning

confidence: 99%

Density Regression and Uncertainty Quantification with Bayesian Deep Noise Neural Networks

Zhang¹,

Liu²,

Kang³

2022

Preprint

View full text Add to dashboard Cite

Deep neural network (DNN) models have achieved state-of-the-art predictive accuracy in a wide range of supervised learning applications. However, accurately quantifying the uncertainty in DNN predictions remains a challenging task. For continuous outcome variables, an even more difficult problem is to estimate the predictive density function, which not only provides a natural quantification of the predictive uncertainty, but also fully captures the random variation in the outcome. In this work, we propose the Bayesian Deep Noise Neural Network (B-DeepNoise), which generalizes standard Bayesian DNNs by extending the random noise variable from the output layer to all hidden layers. The latent random noise equips B-DeepNoise with the flexibility to approximate highly complex predictive distributions and accurately quantify predictive uncertainty. For posterior computation, the unique structure of B-DeepNoise leads to a closed-form Gibbs sampling algorithm that iteratively simulates from the posterior full conditional distributions of the model parameters, circumventing computationally intensive Metropolis-Hastings methods. A theoretical analysis of B-DeepNoise establishes a recursive representation of the predictive distribution and decomposes the predictive variance with respect to the latent parameters. We evaluate B-DeepNoise against existing methods on benchmark regression datasets, demonstrating its superior performance in terms of prediction accuracy, uncertainty quantification accuracy, and uncertainty quantification efficiency. To illustrate our method's usefulness in scientific studies, we apply B-DeepNoise to predict general intelligence from neuroimaging features in the Adolescent Brain Cognitive Development (ABCD) project.

show abstract

Section: Related Workmentioning

confidence: 99%

Density Regression and Uncertainty Quantification with Bayesian Deep Noise Neural Networks

Zhang¹,

Liu²,

Kang³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The potential of stochastic activation layers for UQ was briefly discussed in Lee et al (2019) , but the context of this work was classification tasks, where the predictive uncertainty could already be fully characterized by well-calibrated categorical distributions without using any latent noise. More recently, Sun and Liang (2022) formulated DNNs as latent variable models and included kernel maps in the input layer to avoid feature collinearity. Although the proposed model is capable of UQ, the more challenging problem of DR has not been studied.…”

Section: Introductionmentioning

confidence: 99%

Density regression and uncertainty quantification with Bayesian deep noise neural networks

Zhang

Liu

Kang

2023

Stat

View full text Add to dashboard Cite

Deep neural network (DNN) models have achieved state‐of‐the‐art predictive accuracy in a wide range of applications. However, it remains a challenging task to accurately quantify the uncertainty in DNN predictions, especially those of continuous outcomes. To this end, we propose the Bayesian deep noise neural network (B‐DeepNoise), which generalizes standard Bayesian DNNs by extending the random noise variable from the output layer to all hidden layers. Our model is capable of approximating highly complex predictive density functions and fully learn the possible random variation in the outcome variables. For posterior computation, we provide a closed‐form Gibbs sampling algorithm that circumvents tuning‐intensive Metropolis–Hastings methods. We establish a recursive representation of the predictive density and perform theoretical analysis on the predictive variance. Through extensive experiments, we demonstrate the superiority of B‐DeepNoise over existing methods in terms of density estimation and uncertainty quantification accuracy. A neuroimaging application is included to show our model's usefulness in scientific studies.

show abstract

Magnitude Pruning of Large Pretrained Transformer Models with a Mixture Gaussian Prior

Zhang,

Sun,

Liang

2024

Journal of Data Science

View full text Add to dashboard Cite

Large pretrained transformer models have revolutionized modern AI applications with their state-of-the-art performance in natural language processing (NLP). However, their substantial parameter count poses challenges for real-world deployment. To address this, researchers often reduce model size by pruning parameters based on their magnitude or sensitivity. Previous research has demonstrated the limitations of magnitude pruning, especially in the context of transfer learning for modern NLP tasks. In this paper, we introduce a new magnitude-based pruning algorithm called mixture Gaussian prior pruning (MGPP), which employs a mixture Gaussian prior for regularization. MGPP prunes non-expressive weights under the guidance of the mixture Gaussian prior, aiming to retain the model’s expressive capability. Extensive evaluations across various NLP tasks, including natural language understanding, question answering, and natural language generation, demonstrate the superiority of MGPP over existing pruning methods, particularly in high sparsity settings. Additionally, we provide a theoretical justification for the consistency of the sparse transformer, shedding light on the effectiveness of the proposed pruning method.

show abstract

A Kernel-Expanded Stochastic Neural Network

Cited by 3 publications

References 40 publications

Density Regression and Uncertainty Quantification with Bayesian Deep Noise Neural Networks

Density Regression and Uncertainty Quantification with Bayesian Deep Noise Neural Networks

Density regression and uncertainty quantification with Bayesian deep noise neural networks

Magnitude Pruning of Large Pretrained Transformer Models with a Mixture Gaussian Prior

Contact Info

Product

Resources

About