Language Modeling with Reduced Densities

Abstract

This work originates from the observation that today's state-of-the-art statistical language models are impressive not only for their performance, but also---and quite crucially---because they are built entirely from correlations in unstructured text data. The latter observation prompts a fundamental question that lies at the heart of this paper: What mathematical structure exists in unstructured text data? We put forth enriched category theory as a natural answer. We show that sequences of symbols from a finite alphabet, such as those found in a corpus of text, form a category enriched over probabilities. We then address a second fundamental question: How can this information be stored and modeled in a way that preserves the categorical structure? We answer this by constructing a functor from our enriched category of text to a particular enriched category of reduced density operators. The latter leverages the Loewner order on positive semidefinite operators, which can further be interpreted as a toy example of entailment.

An Enriched Category Theory of Language: From Syntax to Semantics

“…To determine whether [h red , h blue ](c) is empty or not, let's look closer at the functions in (2). The product is computed pointwise, and thus the domain For example, if c is the expression French flag, then there exists a unique natural transformation from h c × h red to h blue if every text that contains both French flag and red as subtexts, also contains blue and so [h red , h blue ](French flag) = * .…”

“…Before going on to the next section, we briefly comment on the work in [2]. In that paper, a [0, 1]-enriched functor is defined between the syntax category L and another [0, 1]-enriched category, embedding L as subcategory of a [0, 1]-enriched category D consisting of "density" operators.…”

“…In that paper, a [0, 1]-enriched functor is defined between the syntax category L and another [0, 1]-enriched category, embedding L as subcategory of a [0, 1]-enriched category D consisting of "density" operators. In this paper we construct a category L of [0, 1]copresheaves on L, and the enriched version of the Yoneda embedding defines an embedding L → L. The reader may think of the [0, 1]-enriched Yoneda embedding as factoring through the functor L → D defined in [2], as in the picture below:…”

“…We define a syntax category, which is a category enriched over the unit-interval [0, 1], that models probability distributions on text continuations [2]. Our semantic category is then defined to be the enriched category of [0, 1]-valued copresheaves on the syntax category.…”

Modelling Lexical Ambiguity with Density Matrices

“…Other applications of density matrices in NLP include modelling entailment in a compositional setting (Balkir et al, 2015;Bankova et al, 2018;Lewis, 2019a;Bradley and Vlassopoulos, 2020). Blacoe et al (2013) also use density matrices to model ambiguity, but in a different setting.…”