Bayesian Inference with Nonlinear Generative Models: Comments on Secure Learning

Abstract

Unlike the classical linear model, nonlinear generative models have been addressed sparsely in the literature. This work aims to bring attention to these models and their secrecy potential. To this end, we invoke the replica method to derive the asymptotic normalized cross entropy in an inverse probability problem whose generative model is described by a Gaussian random field with a generic covariance function. Our derivations further demonstrate the asymptotic statistical decoupling of Bayesian inference algorithms and specify the decoupled setting for a given nonlinear model.The replica solution depicts that strictly nonlinear models establish an all-or-nothing phase transition: There exists a critical load at which the optimal Bayesian inference changes from being perfect to an uncorrelated learning. This finding leads to design of a new secure coding scheme which achieves the secrecy capacity of the wiretap channel. The proposed coding has a significantly smaller codebook size compared to the random coding scheme of Wyner. This interesting result implies that strictly nonlinear generative models are perfectly secured without any secure coding. We justify this latter statement through the analysis of an illustrative model for perfectly secure and reliable inference.

Secure Coding via Gaussian Random Fields

“…In particular, Fyodorov shows that encryption via a purely quadratic Gaussian random field shows an asymptotic phase transition at a threshold signal-to-noise ratio (SNR), below which recovery via the method of leastsquares becomes uncorrelated. Our initial investigations in [16,Section V] shows that combining Fyodorov's encryption with a simple sphere coding technique can achieve a perfect secrecy This work has been accepted for presentation in 2022 IEEE International Symposium on Information Theory (ISIT) in Espoo, Finland. The link to the final version in the Proceedings of ISIT will be available later.…”

“…The derivation of Result 1 relies on the asymptotic characterization of a class Bayesian algorithms used for unsupervised learning in nonlinear generative models. The detailed derivations are given in the extended manuscript [16]. In the sequel, we state a particular form of the generic result in [16] which describes the asymptotic properties of the decoder when it is employed to decode a message encoded via a Gaussian random field.…”

“…The detailed derivations are given in the extended manuscript [16]. In the sequel, we state a particular form of the generic result in [16] which describes the asymptotic properties of the decoder when it is employed to decode a message encoded via a Gaussian random field. This result is then utilized to sketch a proof for Result 1.…”

“…The statistics of the sufficient statistic r are asymptotically described via the results 4 of [16]. We illustrate the asymptotic characterization through the following setting: Consider a vector of uniform bipolar symbols s 0 ∈ {±1} K which is mapped via a Gaussian random field V (•) into x 0 ∈ R N .…”

“…where the posterior distribution p (s 0 |y 0 , V) is determined for the true prior belief on s 0 and the true noise variance σ 2 0 . From the asymptotic results of [16], we can derive the following metrics of this setting in the asymptotic regime, i.e., N, K ↑ ∞ with bounded R = K/N :…”