The Information Bottleneck Problem and its Applications in Machine Learning

Goldfeld, Ziv; Polyanskiy, Yury

doi:10.1109/jsait.2020.2991561

Cited by 111 publications

(94 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We then showed that both IB and PF are closely related to several information-theoretic coding problems such as noisy random coding, hypothesis testing against independence, and dependence dilution. While these connections were partially known in previous work (see e.g., [29,30]), we show that they lead to an improvement on the cardinality of T for computing IB. We then turned our attention to the continuous setting where X and Y are continuous random variables.…”

Section: Summary and Concluding Remarksmentioning

confidence: 55%

“…where R noisy (D) is given in (29). We observed in Section 2.3 that IB is fully characterized by the mapping D → R noisy (D) and thus by A.…”

Section: Theorem A1 ([112]mentioning

confidence: 96%

“…where R = H(Y ) − D, which is equal to IB defined in (4). For more details in connection between noisy source coding and IB, the reader is referred to [27], [28], [91], [93]. Notice that one can study an essentially identical problem where the distortion constraint (28) is replaced by lim n→∞ 1 n I(Y n ; ψ(φ(X n ))) ≥ R, and…”

Section: Operational Meaning Of Ib and Pfmentioning

confidence: 99%

“…Measuring the distortion in the above theorem in terms of the logarithmic loss as in (27), we obtain that…”

Section: Theorem 13 ( [112]mentioning

confidence: 99%

“…We apply this technique to binary X and Y and derive a closed form expression for

– we call this result Mr. Gerber’s Lemma. Relying on the connection between

and noisy source coding [ 19 ] (see [ 29 , 30 ]), we show that the optimal bottleneck variable T in optimization problem ( 2 ) takes values in a set

with cardinality

. Compared to the best cardinality bound previously known (i.e.,

), this result leads to a reduction in the search space’s dimension of the optimization problem ( 2 ) from

.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Bottleneck Problems: An Information and Estimation-Theoretic View

Asoodeh

Calmon

2020

Entropy

View full text Add to dashboard Cite

Information bottleneck (IB) and privacy funnel (PF) are two closely related optimization problems which have found applications in machine learning, design of privacy algorithms, capacity problems (e.g., Mrs. Gerber’s Lemma), and strong data processing inequalities, among others. In this work, we first investigate the functional properties of IB and PF through a unified theoretical framework. We then connect them to three information-theoretic coding problems, namely hypothesis testing against independence, noisy source coding, and dependence dilution. Leveraging these connections, we prove a new cardinality bound on the auxiliary variable in IB, making its computation more tractable for discrete random variables. In the second part, we introduce a general family of optimization problems, termed “bottleneck problems”, by replacing mutual information in IB and PF with other notions of mutual information, namely f-information and Arimoto’s mutual information. We then argue that, unlike IB and PF, these problems lead to easily interpretable guarantees in a variety of inference tasks with statistical constraints on accuracy and privacy. While the underlying optimization problems are non-convex, we develop a technique to evaluate bottleneck problems in closed form by equivalently expressing them in terms of lower convex or upper concave envelope of certain functions. By applying this technique to a binary case, we derive closed form expressions for several bottleneck problems.

show abstract

Section: Summary and Concluding Remarksmentioning

confidence: 55%

“…where R noisy (D) is given in (29). We observed in Section 2.3 that IB is fully characterized by the mapping D → R noisy (D) and thus by A.…”

Section: Theorem A1 ([112]mentioning

confidence: 96%