Learning Document Representations Using Subspace Multinomial Model

Abstract

Subspace multinomial model (SMM) is a log-linear model and can be used for learning low dimensional continuous representation for discrete data. SMM and its variants have been used for speaker verification based on prosodic features and phonotactic language recognition. In this paper, we propose a new variant of SMM that introduces sparsity and call the resulting model as 1 SMM. We show that 1 SMM can be used for learning document representations that are helpful in topic identification or classification and clustering tasks. Our experiments in document classification show that SMM achieves comparable results to models such as latent Dirichlet allocation and sparse topical coding, while having a useful property that the resulting document vectors are Gaussian distributed.

i-Vectors in Language Modeling: An Efficient Way of Domain Adaptation for Feed-Forward Models

“…In this work, we use the l1-SMM variant [11], which adds regularization term to the objective function (3): An l1 regularization on the entries of matrix T and l2 regularization on the i-vectors themselves.…”

“…For details of the training procedure, see the respective paper [11]. We use a publicly available implementation 2 of l1-SMM to obtain i-vectors.…”

“…Both regularization coefficients (l1 on T and l2 on i d ) were set to 10 −4 . We have trained the models using orthant-wise learning [11] until convergence. Figure 3 shows how partial i-vectors evolve in the latent space.…”

“…To address these issues, we propose to enhance FN-LMs by a document summary i-vector estimated from a variant of Subspace Multinomial Model (SMM) [11]. This way, we allow the model to exploit a long context.…”

Hierarchical Transformers for Long Document Classification

“…There are several natural language (NLP) processing tasks that involve such long sequences. Of particular interest are topic identification of spoken conversations [4,5,6] and call center customer satisfaction prediction [7,8,9,10]. Call center conversations, while usually quite short and to the point, often involve agents trying to solve very complex issues that the customers experience, resulting in some calls taking even an hour or more.…”