Distributed Variational Representation Learning

Abstract

The problem of distributed representation learning is one in which multiple sources of information X1, . . . , XK are processed separately so as to learn as much information as possible about some ground truth Y . We investigate this problem from information-theoretic grounds, through a generalization of Tishby's centralized Information Bottleneck (IB) method to the distributed setting. Specifically, K encoders, K ≥ 2, compress their observations X1, . . . , XK separately in a manner such that, collectively, the produced representations preserve as much information as possible about Y . We study both discrete memoryless (DM) and memoryless vector Gaussian data models. For the discrete model, we establish a single-letter characterization of the optimal tradeoff between complexity (or rate) and relevance (or information) for a class of memoryless sources (the observations X1, . . . , XK being conditionally independent given Y ). For the vector Gaussian model, we provide an explicit characterization of the optimal complexity-relevance tradeoff. Furthermore, we develop a variational bound on the complexity-relevance tradeoff which generalizes the evidence lower bound (ELBO) to the distributed setting. We also provide two algorithms that allow to compute this bound: i) a Blahut-Arimoto type iterative algorithm which enables to compute optimal complexity-relevance encoding mappings by iterating over a set of self-consistent equations, and ii) a variational inference type algorithm in which the encoding mappings are parametrized by neural networks and the bound approximated by Markov sampling and optimized with stochastic gradient descent. Numerical results on synthetic and real datasets are provided to support the efficiency of the approaches and algorithms developed in this paper.

Variational Information Bottleneck for Unsupervised Clustering: Deep Gaussian Mixture Embedding

“…We note that although it was assumed that the number of classes was known beforehand (as was the case for almost all competing algorithms in its category), that number could be found (or estimated to within a certain accuracy) through inspection of the resulting bifurcations on the associated information-plane, as was observed for the standard information bottleneck method. Finally, we mention that among the interesting research directions in this line of work, one important question pertains to the distributed learning setting, i.e., along the counterpart, to the unsupervised setting, of the recent work [ 31 , 32 , 33 ], which contained distributed IB algorithms for both discrete and vector Gaussian data models.…”

Reduced-Complexity Optimization of Distributed Quantization Using the Information Bottleneck Principle

“…The equality in (6) holds due to I(X ; Z) = H(X ) − H(X |Z) and the fact that H(X ) can be skipped because it does not depend on the mapping p(z|y). The comparison of ( 5) and ( 6) with (2) reveals that the IB approach is a special formulation of the remote sensing problem using the logarithmic loss functiond(x, z) = − log p(x|z) as a distortion measure whose expectation is H(X |Z) = E X ,Z [− log p(x|z)] [13], [20]. In this case, distortion minimization means maximization of the relevant mutual information I(X ; Z) for given H(X ).…”

“…Although initiated in different areas, a tight connection between the CEO problem and the IB framework exists. For the logarithmic loss function as a distortion measure, the CEO problem can be formulated as a distributed IB problem [20]. Meanwhile, a rich set of IB applications can be found in communications [21]- [26].…”

Information Bottleneck Classification in Extremely Distributed Systems

“…We use the Information Bottleneck (IB) principle presented in [ 6 ] to build the theory behind centralized and decentralized classification models. The analysis of the supervised and unsupervised information bottleneck problems was performed in [ 23 ] and generalized to the distributed setup in [ 24 ]. In this work, we extend the IBN to demonstrate the importance of compression in the form of vector quantization for the classification problem.…”