An Information Theoretic View on Learning of Artificial Neural Networks

Balda, Emilio Rafael; Behboodi, Arash; Mathar, Rudolf

doi:10.1109/icspcs.2018.8631758

Cited by 6 publications

(7 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The convolutional neural network could also be investigated regarding whether it satisfies DPI or otherwise. In the case of a feed-forward neural network, the Markovian structure and data processing inequalities across layers are generally accepted [ 10 , 12 ]. In a previous study [ 9 ], it was stated that this could also be seen in CNN networks, despite the calculation limits.…”

Section: Experiments and Resultsmentioning

confidence: 99%

“…In addition to these, because the selection of hyperparameters, such as the learning speed and batch size is also intuitive and has no transparent algorithmic structure, these networks are not reproducible. Thus, efforts [ 8 , 12 ] are being made to make DNNs more understandable and transparent.…”

Section: Related Workmentioning

confidence: 99%

“…Moreover, Yu et al have introduced a new matrix-based Renyi’s α-entropy technique to analyse the information flow in stacked autoencoders and CNNs, respectively [ 9 , 21 ]. Furthermore, Balda et al have adopted the information theoretic method and incorporated it with generalisation errors and suggesting the learning process of neural networks [ 12 ].…”

Section: Related Workmentioning

confidence: 99%

“…We are motivated by the studies on deep learning and information bottleneck theory by Tishby et al [ 10 , 11 , 12 ]. In this study, mutual information across different layers of CNN is used to understand the effects of network parameters, such as kernel size, number of convolutional layers, dropout layer and dropout rate, on the learning process.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Evaluating the Learning Procedure of CNNs through a Sequence of Prognostic Tests Utilising Information Theoretical Measures

Shi

Silva

Aslan

et al. 2021

Entropy

View full text Add to dashboard Cite

Deep learning has proven to be an important element of modern data processing technology, which has found its application in many areas such as multimodal sensor data processing and understanding, data generation and anomaly detection. While the use of deep learning is booming in many real-world tasks, the internal processes of how it draws results is still uncertain. Understanding the data processing pathways within a deep neural network is important for transparency and better resource utilisation. In this paper, a method utilising information theoretic measures is used to reveal the typical learning patterns of convolutional neural networks, which are commonly used for image processing tasks. For this purpose, training samples, true labels and estimated labels are considered to be random variables. The mutual information and conditional entropy between these variables are then studied using information theoretical measures. This paper shows that more convolutional layers in the network improve its learning and unnecessarily higher numbers of convolutional layers do not improve the learning any further. The number of convolutional layers that need to be added to a neural network to gain the desired learning level can be determined with the help of theoretic information quantities including entropy, inequality and mutual information among the inputs to the network. The kernel size of convolutional layers only affects the learning speed of the network. This study also shows that where the dropout layer is applied to has no significant effects on the learning of networks with a lower dropout rate, and it is better placed immediately after the last convolutional layer with higher dropout rates.

show abstract

Section: Experiments and Resultsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Evaluating the Learning Procedure of CNNs through a Sequence of Prognostic Tests Utilising Information Theoretical Measures

Shi

Silva

Aslan

et al. 2021

Entropy

View full text Add to dashboard Cite

show abstract

“…Shwartz-Ziv and Tishby [10]'s information bottleneck theory approach as a theoretical basis for explaining deep neural network learning has triggered much controversy. These discussions have focused on attributing the causes of the observed compression phase to stochasticity in training [10,38,39], the specific activation function used [12], initialisation of models [40] and the method used for estimating mutual information [41]. Fundamental to much of the discussion in recent literature on this subject is the difficulty of estimating mutual information in high dimensional datasets, which requires accurate models of the probability distributions.…”

Section: Discussionmentioning

confidence: 99%

Information Bottleneck Theory Based Exploration of Cascade Learning

Farrahi

Niranjan

2021

Entropy

View full text Add to dashboard Cite

In solving challenging pattern recognition problems, deep neural networks have shown excellent performance by forming powerful mappings between inputs and targets, learning representations (features) and making subsequent predictions. A recent tool to help understand how representations are formed is based on observing the dynamics of learning on an information plane using mutual information, linking the input to the representation (I(X;T)) and the representation to the target (I(T;Y)). In this paper, we use an information theoretical approach to understand how Cascade Learning (CL), a method to train deep neural networks layer-by-layer, learns representations, as CL has shown comparable results while saving computation and memory costs. We observe that performance is not linked to information–compression, which differs from observation on End-to-End (E2E) learning. Additionally, CL can inherit information about targets, and gradually specialise extracted features layer-by-layer. We evaluate this effect by proposing an information transition ratio, I(T;Y)/I(X;T), and show that it can serve as a useful heuristic in setting the depth of a neural network that achieves satisfactory accuracy of classification.

show abstract