Language as a Latent Variable: Discrete Generative Models for Sentence Compression

Abstract

In this work we explore deep generative models of text in which the latent representation of a document is itself drawn from a discrete language model distribution. We formulate a variational auto-encoder for inference in this model and apply it to the task of compressing sentences. In this application the generative model first draws a latent summary sentence from a background language model, and then subsequently draws the observed sentence conditioned on this latent summary. In our empirical evaluation we show that generative formulations of both abstractive and extractive compression yield state-of-the-art results when trained on a large amount of supervised data. Further, we explore semi-supervised compression scenarios where we show that it is possible to achieve performance competitive with previously proposed supervised models while training on a fraction of the supervised data.

Semi-supervised Structured Prediction with Neural CRF Autoencoder

“…The difficulty of providing sufficient supervision has motivated work on semi-supervised and unsupervised learning for many of these tasks (McClosky et al, 2006;Spitkovsky et al, 2010;Subramanya et al, 2010;Stratos and Collins, 2015;Marinho et al, 2016;Tran et al, 2016), including several that also used autoencoders (Ammar et al, 2014;Lin et al, 2015;Miao and Blunsom, 2016;Kociský et al, 2016;Cheng et al, 2017). In this paper we expand on these works, and suggest a neural CRF autoencoder, that can leverage both labeled and unlabeled data.…”

Morphological Inflection Generation with Multi-space Variational Encoder-Decoders

“…At the early stage of training, we set λ to be zero and let the model first figure out how to project the representation of the source sequence to a roughly right point in the space and then regularize it with the KL cost. This technique can also be seen in (Kočiskỳ et al, 2016;Miao and Blunsom, 2016). Input Dropout in the Decoder: Besides annealing the KL cost, we also randomly drop out the input token with a probability of β at each time step of the decoder during learning.…”

Proceedings of the CoNLL SIGMORPHON 2017 Shared Task: Universal Morphological Reinflection

“…At the early stage of training, we set λ to be zero and let the model first figure out how to project the representation of the source sequence to a roughly right point in the space and then regularize it with the KL cost. This technique can also be seen in (Kočiskỳ et al, 2016;Miao and Blunsom, 2016). Input Dropout in the Decoder: Besides annealing the KL cost, we also randomly drop out the input token with a probability of β at each time step of the decoder during learning.…”