Tensor networks and efficient descriptions of classical data

Lu, Sirui; Kanász-Nagy, Márton; Kukuljan, Ivan; Cirac, J. Ignacio

doi:10.48550/arxiv.2103.06872

Cited by 14 publications

(15 citation statements)

References 92 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…bility to a given learning task via analyzing the entanglement properties [62,63]. In addition, tensor networks provide a convenient framework to study how and why certain quantum learning models would exhibit exponential advantages over their classical analogues [25,31,64].…”

mentioning

confidence: 99%

The Presence and Absence of Barren Plateaus in Tensor-network Based Machine Learning

Liu,

Yu,

Duan

et al. 2021

Preprint

View full text Add to dashboard Cite

Tensor networks are efficient representations of high-dimensional tensors with widespread applications in quantum many-body physics. Recently, they have been adapted to the field of machine learning, giving rise to an emergent research frontier that has attracted considerable attention. Here, we study the trainability of tensor-network based machine learning models by exploring the landscapes of different loss functions, with a focus on the matrix product states (also called tensor trains) architecture. In particular, we rigorously prove that barren plateaus (i.e., exponentially vanishing gradients) prevail in the training process of the machine learning algorithms with global loss functions. Whereas, for local loss functions the gradients with respect to variational parameters near the local observables do not vanish as the system size increases. Therefore, the barren plateaus are absent in this case and the corresponding models could be efficiently trainable. Our results reveal a crucial aspect of tensor-network based machine learning in a rigorous fashion, which provide a valuable guide for both practical applications and theoretical studies in the future.

show abstract

mentioning

confidence: 99%

The Presence and Absence of Barren Plateaus in Tensor-network Based Machine Learning

Liu,

Yu,

Duan

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…The EE provides an upper bound on mutual information, an important information-theoretic properties characterizing the data. Recently mutual information and the entanglement entropy of data in the training set have been studied in [21,23,24,25]. Our work provides an alternative conceptually better way to evaluate it, as it is based on the full set, rather than the training set alone.…”

Section: Resultsmentioning

confidence: 99%

Tensor network to learn the wavefunction of data

Dymarsky¹,

Pavlenko²

2021

Preprint

View full text Add to dashboard Cite

How many different ways are there to handwrite digit 3? To quantify this question imagine extending a dataset of handwritten digits MNIST by sampling additional images until they start repeating. We call the collection of all resulting images of digit 3 the "full set." To study the properties of the full set we introduce a tensor network architecture which simultaneously accomplishes both classification (discrimination) and sampling tasks. Qualitatively, our trained network represents the indicator function of the full set. It therefore can be used to characterize the data itself. We illustrate that by studying the full sets associated with the digits of MNIST. Using quantum mechanical interpretation of our network we characterize the full set by calculating its entanglement entropy. We also study its geometric properties such as mean Hamming distance, effective dimension, and size. The latter answers the question above -the total number of black and white threes written MNIST style is 2 72 .

show abstract

“…It would be interesting to generalize the current analysis to deep (multi-layer) RACs models [36] to see the interplay between long-range information propagation in the recurrent networks and the meaningful word embedding in other realistic natural language processing tasks, such as sequence to sequence modeling. Perhaps one could also find a minimal deep RACs model that reproduces the power-law decay in the mutual information between characters, which is a feature of classical English texts [40,41]. Recently, variants of standard many-body quantum states have been analyzed as highly expressive variational ansatz to estimate probability distribution [13,19,42]; it'd also be interesting to implement such models for realistic natural language processing tasks and investigate how word embedding could help boost models prediction accuracy.…”

Section: Discussion and Outlookmentioning

confidence: 99%

Explainable Natural Language Processing with Matrix Product States

Tangpanitanon,

Mangkang,

Bhadola

et al. 2021

Preprint

View full text Add to dashboard Cite

Despite empirical successes of recurrent neural networks (RNNs) in natural language processing (NLP), theoretical understanding of RNNs is still limited due to intrinsically complex computations in RNNs. We perform a systematic analysis of RNNs' behaviors in a ubiquitous NLP task, the sentiment analysis of movie reviews, via the mapping between a class of RNNs called recurrent arithmetic circuits (RACs) and a matrix product state (MPS). Using the von-Neumann entanglement entropy (EE) as a proxy for information propagation, we show that single-layer RACs possess a maximum information propagation capacity, reflected by the saturation of the EE. Enlarging the bond dimension of an MPS beyond the EE saturation threshold does not increase the prediction accuracies, so a minimal model that best estimates the data statistics can be constructed. Although the saturated EE is smaller than the maximum EE achievable by the area law of an MPS, our model achieves ∼ 99% training accuracies in realistic sentiment analysis data sets. Thus, low EE alone is not a warrant against the adoption of single-layer RACs for NLP. Contrary to a common belief that longrange information propagation is the main source of RNNs' expressiveness, we show that single-layer RACs also harness high expressiveness from meaningful word vector embeddings. Our work sheds light on the phenomenology of learning in RACs and more generally on the explainability aspects of RNNs for NLP, using tools from many-body quantum physics.

show abstract

Tensor networks and efficient descriptions of classical data

Cited by 14 publications

References 92 publications

The Presence and Absence of Barren Plateaus in Tensor-network Based Machine Learning

The Presence and Absence of Barren Plateaus in Tensor-network Based Machine Learning

Tensor network to learn the wavefunction of data

Explainable Natural Language Processing with Matrix Product States

Contact Info

Product

Resources

About