Deep learning and the information bottleneck principle

Tishby, Naftali; Zaslavsky, Noga

doi:10.1109/itw.2015.7133169

Cited by 1,139 publications

(1,027 citation statements)

References 11 publications

Supporting

Mentioning

1,010

Contrasting

Unclassified

Order By: Relevance

“…More recently, there is a small line of emerging work investigating the behaviour of neural networks from an information-theoretic perspective [15][16][17][18][19][20], with some work going as far back as [21]. The most relevant of these is the work by Schwartz-Ziv and Tishby [16], who show that feed-forward deep neural networks undergo a dynamic transition between drift-and diffusion-like regimes during training.…”

Section: Related Workmentioning

confidence: 99%

The Partial Information Decomposition of Generative Neural Network Models

Tax¹,

Mediano

Shanahan

2017

Entropy

View full text Add to dashboard Cite

In this work we study the distributed representations learnt by generative neural network models. In particular, we investigate the properties of redundant and synergistic information that groups of hidden neurons contain about the target variable. To this end, we use an emerging branch of information theory called partial information decomposition (PID) and track the informational properties of the neurons through training. We find two differentiated phases during the training process: a first short phase in which the neurons learn redundant information about the target, and a second phase in which neurons start specialising and each of them learns unique information about the target. We also find that in smaller networks individual neurons learn more specific information about certain features of the input, suggesting that learning pressure can encourage disentangled representations.

show abstract

Section: Related Workmentioning

confidence: 99%

The Partial Information Decomposition of Generative Neural Network Models

Tax¹,

Mediano

Shanahan

2017

Entropy

View full text Add to dashboard Cite

show abstract

“…30 Conventionally, the process of generalizing the performance of the classifier for eventual new data 31 requires a series of good-practices in the use of the available data to train and then evaluate it [2,3]. In 32 this supervised scheme, the evaluation of the performance of the classifier involves the comparison of 33 the true labels K vs. the predicted labelsK, as the abstracted diagram in Fig.…”

Section: (B)mentioning

confidence: 99%

Assessing Information Transmission in Data Transformations with the Channel Multivariate Entropy Triangle

Valverde-Albacete

Peláez-Moreno

2018

Entropy

View full text Add to dashboard Cite

Data transformation, e.g., feature transformation and selection, is an integral part of any machine learning procedure. In this paper, we introduce an information-theoretic model and tools to assess the quality of data transformations in machine learning tasks. In an unsupervised fashion, we analyze the transformation of a discrete, multivariate source of information X into a discrete, multivariate sink of information Y related by a distribution P XY . The first contribution is a decomposition of the maximal potential entropy of (X, Y), which we call a balance equation, into its (a) non-transferable, (b) transferable, but not transferred, and (c) transferred parts. Such balance equations can be represented in (de Finetti) entropy diagrams, our second set of contributions. The most important of these, the aggregate channel multivariate entropy triangle, is a visual exploratory tool to assess the effectiveness of multivariate data transformations in transferring information from input to output variables. We also show how these decomposition and balance equations also apply to the entropies of X and Y, respectively, and generate entropy triangles for them. As an example, we present the application of these tools to the assessment of information transfer efficiency for Principal Component Analysis and Independent Component Analysis as unsupervised feature transformation and selection procedures in supervised classification tasks.

show abstract

“…Design of the optimal distributed quantizer for the above scenario has been considered by the authors of [13][14][15][16][17][18], who suggest cyclic algorithms based on alternating minimization [19], to find the optimal quantization rules. The algorithm starts by initial guesses about the quantizers, that is, 0 1 , .…”

Section: International Journal Of Distributed Sensor Networkmentioning

confidence: 99%

Distributed Channel-Aware Quantization Based on Maximum Mutual Information

Movaghati

Ardakani

2016

International Journal of Distributed Sensor Networks

View full text Add to dashboard Cite

In distributed sensing systems with constrained communication capabilities, sensors' noisy measurements must be quantized locally before transmitting to the fusion centre. When the same parameter is observed by a number of sensors, the local quantization rules must be jointly designed to optimize a global objective function. In this work we jointly design the local quantizers by maximizing the mutual information as the optimization criterion, so that the quantized measurements carry the most information about the unknown parameter. A low-complexity iterative approach is suggested for finding the local quantization rules. Using the mutual information as the design criterion, we can easily integrate the effect of communication channels in the design and consequently design channel-aware quantization rules. We observe that the optimal design depends on both the measurement and channel noises. Moreover, our algorithm can be used to design quantizers that can be deployed in different applications. We demonstrate the success of our technique through simulating estimation and detection applications, where our method achieves estimation and detection errors as low as when designing for those special purposes.

show abstract

Deep learning and the information bottleneck principle

Cited by 1,139 publications

References 11 publications

The Partial Information Decomposition of Generative Neural Network Models

The Partial Information Decomposition of Generative Neural Network Models

Assessing Information Transmission in Data Transformations with the Channel Multivariate Entropy Triangle

Distributed Channel-Aware Quantization Based on Maximum Mutual Information

Contact Info

Product

Resources

About