Improving Bayesian Neural Networks by Adversarial Sampling

Abstract

Bayesian neural networks (BNNs) have drawn extensive interest due to the unique probabilistic representation framework. However, Bayesian neural networks have limited publicized deployments because of the relatively poor model performance in real-world applications. In this paper, we argue that the randomness of sampling in Bayesian neural networks causes errors in the updating of model parameters during training and some sampled models with poor performance in testing. To solve this, we propose to train Bayesian neural networks with Adversarial Distribution as a theoretical solution. To avoid the difficulty of calculating Adversarial Distribution analytically, we further present the Adversarial Sampling method as an approximation in practice. We conduct extensive experiments with multiple network structures on different datasets, e.g., CIFAR-10 and CIFAR-100. Experimental results validate the correctness of the theoretical analysis and the effectiveness of the Adversarial Sampling on improving model performance. Additionally, models trained with Adversarial Sampling still keep their ability to model uncertainties and perform better when predictions are retained according to the uncertainties, which further verifies the generality of the Adversarial Sampling approach.

A Novel Sybil Attack Detection Scheme in MobileIoT Based on Collaborate Edge Computing

“…Sybil attack is a kind of Byzantine attack, and there are other works against Byzantine attacks in the AI field. Work [28] presented an alarming mechanism to build a Byzantine-robust federated learning system, Work [29] train a Bayesian neural network via an adversarial distribution to improve the practical applications' performance. However, work [28] is aimed at federated learning, and work [29] is more about theoretical research, which is not IoT.…”

A novel Sybil attack detection scheme in mobile IoT based on collaborate edge computing

“…Sybil attack is a kind of Byzantine attack, and there are other works against Byzantine attacks in the artificial intelligence field. Work [22] presented an alarming mechanism to build a Byzantine-robust federated learning system, and work [23] trains a Bayesian neural network via an adversarial distribution to improve the practical applications' performance. However, work [22] is aimed at federated learning, and work [23] is more about theoretical research, which is not IoT.…”

Towards Fast and Stable Federated Learning: Confronting Heterogeneity via Knowledge Anchor

“…The personalized federated learning allows each client to possess its own model and utilize the global aggregated information to assist its training [20,21,22]. SOTA personalized federated learning methods adopt various strategies to personalize their local model, including fine-tuning global model [23], splitting and personalizing the projection heads [24], personalized aggregation [25,26,27], additional personalized models [22,28], etc. Though personalized federated models demonstrate great performance improvement in heterogeneous data, the generic performance and the generalization ability of these models are limited.…”