Learnability for the Information Bottleneck

Abstract

The Information Bottleneck (IB) method (Tishby et al. (2000)) provides an insightful and principled approach for balancing compression and prediction for representation learning. The IB objective I(X; Z) − βI(Y ; Z) employs a Lagrange multiplier β to tune this trade-off. However, in practice, not only is β chosen empirically without theoretical guidance, there is also a lack of theoretical understanding between β, learnability, the intrinsic nature of the dataset and model capacity. In this paper, we show that if β is improperly chosen, learning cannot happen -the trivial representation P (Z|X) = P (Z) becomes the global minimum of the IB objective. We show how this can be avoided, by identifying a sharp phase transition between the unlearnable and the learnable which arises as β is varied. This phase transition defines the concept of IB-Learnability. We prove several sufficient conditions for IB-Learnability, which provides theoretical guidance for choosing a good β. We further show that IB-learnability is determined by the largest confident, typical, and imbalanced subset of the examples (the conspicuous subset), and discuss its relation with model capacity. We give practical algorithms to estimate the minimum β for a given dataset. We also empirically demonstrate our theoretical conditions with analyses of synthetic datasets, MNIST, and CIFAR10.• We show that improperly chosen β may result in a failure to learn: the trivial solution P (Z|X) = P (Z) becomes the global minimum of the IB objective, even for β 1 (Section 1.1).• We introduce the concept of IB-Learnability, and show that when we vary β, the IB objective will un-arXiv:1907.07331v1 [cs.LG]

The Convex Information Bottleneck Lagrangian

“…The range of the Lagrange multipliers that allow the exploration of the IB curve is contained by which is also contained by , where where is the derivative of w.r.t. evaluated at r, is the set of possible realizations of X and and are defined as in [ 27 ] (Note in [ 27 ] they consider the dual problem (see Appendix G ), so when they refer to it translates to β in this article). That is, .…”

“…Corollaries 2 and 3 allow us to reduce the range search for when we want to explore the IB curve. Practically, might be difficult to calculate so Wu et al [ 27 ] derived an algorithm to approximate it. However, we still recommend setting the numerator to 1 for simplicity.…”

“…The main difference comes from the discontinuities in performance for increasing , which cause is still unknown (cf. Wu et al [ 27 ]). It has been observed, however, that the bottleneck variable performs an intrinsic clusterization in classification tasks (see, for instance, [ 21 , 26 , 42 ] or Figure 2 b).…”

“…The Lagrange multiplier selection is important since (i) sometimes even choices of lead to trivial representations such that , and (ii) there exist some discontinuities on the performance level w.r.t. the values of [ 27 ].…”

Optimal prediction with resource constraints using the information bottleneck

“…This demonstrates the existence of a critical β for each predictive coding scheme, above which m needs to be increased to extract more predictive information and below which additional values of the representation variable encode redundant portions of allele frequency space. While we do not estimate the critical β, approaches to estimating them are presented in [42,43].…”