Dissecting Deep Learning Networks—Visualizing Mutual Information

Abstract

Deep Learning (DL) networks are recent revolutionary developments in artificial intelligence research. Typical networks are stacked by groups of layers that are further composed of many convolutional kernels or neurons. In network design, many hyper-parameters need to be defined heuristically before training in order to achieve high cross-validation accuracies. However, accuracy evaluation from the output layer alone is not sufficient to specify the roles of the hidden units in associated networks. This results in a significant knowledge gap between DL’s wider applications and its limited theoretical understanding. To narrow the knowledge gap, our study explores visualization techniques to illustrate the mutual information (MI) in DL networks. The MI is a theoretical measurement, reflecting the relationship between two sets of random variables even if their relationship is highly non-linear and hidden in high-dimensional data. Our study aims to understand the roles of DL units in classification performance of the networks. Via a series of experiments using several popular DL networks, it shows that the visualization of MI and its change patterns between the input/output with the hidden layers and basic units can facilitate a better understanding of these DL units’ roles. Our investigation on network convergence suggests a more objective manner to potentially evaluate DL networks. Furthermore, the visualization provides a useful tool to gain insights into the network performance, and thus to potentially facilitate the design of better network architectures by identifying redundancy and less-effective network units.

Learning in Feedforward Neural Networks Accelerated by Transfer Entropy

“…We will refer to them in the following. TE has been used for the quantification of effective connectivity between neurons [10,11,12,13]. To the extent of our knowledge, the work in [14,15] represent the only attempts to use TE for improving the learning capability of neural networks.…”

Enabling the Sense of Self in a Dual-Arm Robot

“…To further understand whether our Level 1 architecture learns to differentiate the robot from the environment, we computed the Mutual Information [24] for each group's train dataset (Table I). Our objective is to measure and compare if four Level 1 trained architectures have a degree of similar knowledge that it is invariant to the training set.…”

Mutual Information Based Learning Rate Decay for Stochastic Gradient Descent Training of Deep Neural Networks

“…This paper presents significant improvements over the previous version, significant change to the LR policy to enable it to build on the well understood convergence properties of decaying LR SGD and significantly better experimental results. The use of MI as a metric for LR adaptation is intended to lead to further work towards a deeper understanding of DNNs [ 19 ] and learning of DNNs through maximizing Mutual Information [ 16 , 20 ].…”