Deep Learning and Quantum Entanglement: Fundamental Connections with Implications to Network Design

Abstract

Deep convolutional networks have witnessed unprecedented success in various machine learning applications. Formal understanding on what makes these networks so successful is gradually unfolding, but for the most part there are still significant mysteries to unravel. The inductive bias, which reflects prior knowledge embedded in the network architecture, is one of them. In this work, we establish a fundamental connection between the fields of quantum physics and deep learning. We use this connection for asserting novel theoretical observations regarding the role that the number of channels in each layer of the convolutional network fulfills in the overall inductive bias. Specifically, we show an equivalence between the function realized by a deep convolutional arithmetic circuit (ConvAC) and a quantum many-body wave function, which relies on their common underlying tensorial structure. This facilitates the use of quantum entanglement measures as welldefined quantifiers of a deep network's expressive ability to model intricate correlation structures of its inputs. Most importantly, the construction of a deep convolutional arithmetic circuit in terms of a Tensor Network is made available. This description enables us to carry a graph-theoretic analysis of a convolutional network, tying its expressiveness to a min-cut in the graph which characterizes it. Thus, we demonstrate a direct control over the inductive bias of the designed deep convolutional network via its channel numbers, which we show to be related to the min-cut in the underlying graph. This result is relevant to any practitioner designing a convolutional network for a specific task. We theoretically analyze convolutional arithmetic circuits, and empirically validate our findings on more common convolutional networks which involve ReLU activations and max pooling. Beyond the results described above, the description of a deep convolutional network in well-defined graph-theoretic tools and the formal structural connection to quantum entanglement, are two interdisciplinary bridges that are brought forth by this work.

Magic entanglement renormalization for quantum fields

“…Tensor networks [1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16] offer an efficient representation of otherwise exponentially complex objects, such as the manybody wavefunction of a quantum system. Over the last two decades, tensor networks have become of importance in an ever growing number of research areas, including quantum information and condensed matter [1][2][3][4][5][6][7][8][9][10][11], quantum chemistry [17][18][19][20], statistical mechanics [21][22][23][24], quantum gravity [25][26][27][28][29], and machine learning [30][31][32][33][34]. Two particularly useful tensor networks are the matrix product state (MPS) [1][2][3]…”

Entangled q-Convolutional Neural Nets

“…The q-CNN architecture discussed in this work is based on the theoretical architecture, named deep convolutional arithmetic circuit, introduced in [23,24]. The product pooling proposed in [23,24] presents a challenge in training the network described above, since it can easily lead to numerical instabilities such as underflow or overflow. We would like to render the network trainable in practice, and do so without spoiling the analogy to quantum many-body systems.…”

“…In [31], the entanglement entropy of the final trained network was computed for a very different architecture directly related to tensor networks, but not its evolution during training and the correlation between entanglement and accuracy. In [32] and [24], the possibility was mentioned that the requirement of being capable to accommodate the entanglement could be used to guide the design of the network. We note that the values of the entanglement entropy in our experiments on the MNIST and F-MNIST datasets is of the order of log 2.…”

“…As mentioned before, without batch normalisation, the architecture consisting of the specific form of convolutional and product pooling layers described above is the convolutional arithmetic circuit that the authors proposed in [23], and further studied in [24,33]. It was also pointed out that such an architecture implements a (hierarchical) Tucker decomposition of the network tensors.…”

“…Inspired by these questions, there have been increasing efforts to build a bridge between the two fields of machine learning and quantum many-body systems, and in particular tensor networks [9][10][11][12][13][14][15][16][17][18][19][20][21][22][23][24][25]. In this work, we continue to strengthen this bridge, focusing on convolutional neural networks (CNNs).…”