Compression-Based Regularization With an Application to Multitask Learning

Abstract

This paper investigates, from information theoretic grounds, a learning problem based on the principle that any regularity in a given dataset can be exploited to extract compact features from data, i.e., using fewer bits than needed to fully describe the data itself, in order to build meaningful representations of a relevant content (multiple labels). We begin by introducing the noisy lossy source coding paradigm with the log-loss fidelity criterion which provides the fundamental tradeoffs between the cross-entropy loss (average risk) and the information rate of the features (model complexity). Our approach allows an information theoretic formulation of the multi-task learning (MTL) problem which is a supervised learning framework in which the prediction models for several related tasks are learned jointly from common representations to achieve better generalization performance. Then, we present an iterative algorithm for computing the optimal tradeoffs and its global convergence is proven provided that some conditions hold. An important property of this algorithm is that it provides a natural safeguard against overfitting, because it minimizes the average risk taking into account a penalization induced by the model complexity. Remarkably, empirical results illustrate that there exists an optimal information rate minimizing the excess risk which depends on the nature and the amount of available training data. An application to hierarchical text categorization is also investigated, extending previous works.

Information Bottleneck and Representation Learning

“…It is easy to show that this region corresponds to the set of achievable values of relevance and rate (µ, R) for the corresponding noisy lossy source coding problem with logarithmic distortion as was defined in Section 1.3.3. This set is closed and convex and it is not difficult to show that [34]:…”

Generalization Error in Deep Learning

“…Therefore, it is of interest to explore links between information theory, representation learning and the information bottleneck, in order to cast insights onto the performance of deep neural networks under an information-theoretic lens. Preliminary steps in this direction are taken in [65,66].…”

New Results on Testing Against Independence with Rate-Limited Constraints

“…An important part of this algorithm requires to compute ξ(R), whose solution involves an optimization formulation with respect to the encoder f n and the rate R [26]. For the computation of ξ(R) we use the algorithm presented in [29] which is a generalization of the Blahut-Arimoto algorithm [30]. 3 Blocklength Table I, II and III show the values of (UB( n , R) − LB( n , R)).…”

“…Under some mild conditions given in[29], there exist guarantees to ensure this optimization converges to ξ(R).…”