Learning in Feedforward Neural Networks Accelerated by Transfer Entropy

Abstract

Current neural networks architectures are many times harder to train because of the increasing size and complexity of the used datasets. Our objective is to design more efficient training algorithms utilizing causal relationships inferred from neural networks. The transfer entropy (TE) was initially introduced as an information transfer measure used to quantify the statistical coherence between events (time series). Later, it was related to causality, even if they are not the same. There are only few papers reporting applications of causality or TE in neural networks. Our contribution is an information-theoretical method for analyzing information transfer between the nodes of feedforward neural networks. The information transfer is measured by the TE of feedback neural connections. Intuitively, TE measures the relevance of a connection in the network and the feedback amplifies this connection. We introduce a backpropagation type training algorithm that uses TE feedback connections to improve its performance.

Learning in Convolutional Neural Networks Accelerated by Transfer Entropy

“…In our study, we add specific TE feedback connections between neurons to improve performance. Our results confirm what we obtained in our previous study on a simple feedforward neural architecture [ 20 ]. Adding the TE feedback parameter accelerates the training process, as fewer epochs are needed.…”

“…This can be related to the hierarchy of quickly-changing vs. slowly-changing parameters in learning neural causal models [ 45 ]. We observed that the TE feedback generates stability during training, this being compliant with the results presented in [ 20 ].…”

“…We extend here the results from [ 20 ] and adapt them to a much larger neural architecture—the CNN network. Rather then being a simple generalization, it is a novel approach, as we had to redefine the network training process.…”

“…Compared with a standard backpropagation training algorithm, the addition of the TE feedback in the training scheme implies a computational overhead needed to compute the TE values in the training stage. However, according to our experiments, adding the TE feedback parameter has three benefits [ 20 ]: (a) it accelerates the training process—in general, less epochs are needed; (b) generally achieve a better test set accuracy; and (c) it generates stability during training. The neural models trained in [ 20 ] were relatively small.…”

“…However, according to our experiments, adding the TE feedback parameter has three benefits [ 20 ]: (a) it accelerates the training process—in general, less epochs are needed; (b) generally achieve a better test set accuracy; and (c) it generates stability during training. The neural models trained in [ 20 ] were relatively small. This allowed to use all training samples to compute the TE from time series.…”