Concentration of the Matrix-Valued Minimum Mean-Square Error in Optimal Bayesian Inference

Abstract

We consider Bayesian inference of signals with vector-valued entries. Extending concentration techniques from the mathematical physics of spin glasses, we show that the matrix-valued minimum mean-square error concentrates when the size of the problem increases. Such results are often crucial for proving single-letter formulas for the mutual information when they exist. Our proof is valid in the optimal Bayesian inference setting, meaning that it relies on the assumption that the model and all its hyper-parameters are known. Examples of inference and learning problems covered by our results are spiked matrix and tensor models, the committee machine neural network with few hidden neurons in the teacher-student scenario, or multi-layers generalized linear models.

An inference problem in a mismatched setting: a spin-glass model with Mattis interaction

“…From those one can also show that all these models fulfill a set of identities and correlation inequalities [3][4][5] that imply the peculiar feature of the emerging thermodynamics, known as replica symmetry, i.e. the system properties are fully encoded in a self-averaging quantity, the overlap [6][7][8].…”

An inference problem in a mismatched setting: a spin-glass model with Mattis interaction

“…From those one can also show that all these models fulfill a set of identities and correlation inequalities [7,8,19] that imply the peculiar feature of the emerging thermodynamics, known as replica symmetry, i.e. the system properties are fully encoded in a self-averaging quantity, the overlap [29,33,34].…”