A Quantum Many-body Wave Function Inspired Language Modeling Approach

Abstract

The recently proposed quantum language model (QLM) aimed at a principled approach to modeling term dependency by applying the quantum probability theory. The latest development for a more effective QLM has adopted word embeddings as a kind of global dependency information and integrated the quantum-inspired idea in a neural network architecture. While these quantum-inspired LMs are theoretically more general and also practically effective, they have two major limitations. First, they have not taken into account the interaction among words with multiple meanings, which is common and important in understanding natural language text. Second, the integration of the quantum-inspired LM with the neural network was mainly for effective training of parameters, yet lacking a theoretical foundation accounting for such integration. To address these two issues, in this paper, we propose a Quantum Many-body Wave Function (QMWF) inspired language modeling approach. The QMWF inspired LM can adopt the tensor product to model the aforesaid interaction among words. It also enables us to reveal the inherent necessity of using Convolutional Neural Network (CNN) in QMWF language modeling. Furthermore, our approach delivers a simple algorithm to represent and match text/sentence pairs. Systematic evaluation shows the effectiveness of the proposed QMWF-LM algorithm, in comparison with the state of the art quantum-inspired LMs and a couple of CNN-based methods, on three typical Question Answering (QA) datasets. KEYWORDSLanguage modeling, quantum many-body wave function, convolutional neural network ACM Reference Format:

Semantic Hilbert Space for Text Representation Learning

“…The Semantic Hilbert Space represents different levels of semantic units, ranging from basic sememes, words and sentences, on a unified complexvalued vector space. This is fundamentally different from existing quantum-inspired neural networks for question answering [31,32] which are based on a real vector space. In addition, we introduce a new semantic abstraction, named as Semantic Measurements, which are also embedded in the same vector space and trainable to extract high-level features from the mixed system.…”

A Survey of Quantum Theory Inspired Approaches to Information Retrieval

“…A Quantum many body wave function based language model is presented in [Zhang et al 2018d given word is written as a tensor product of the state vectors in all the Hilbert spaces. A complex phrase is ascribed to the state vector.…”

“…In [Zhang et al 2018d], a quantum many-body wave function is used to model the semantic meaning of words within a local context, i.e., sentences, and a global context, i.e., corpus. In particular, each word is represented by different base states in the sense that each basis corresponds to a different word meaning.…”

A Quantum Expectation Value Based Language Model with Application to Question Answering

“…Our ablation studies show that different observables can dramatically affect the model’s performance, and the off-diagonal elements corresponding the semantic superpositions significantly boost the model’s performance. Therefore, it would be interesting to further explore the possible observables and other approaches to construct superpositions [ 36 ].…”