Generating Sentences from a Continuous Space

Bowman, Samuel R.; Vilnis, Luke; Vinyals, Oriol; Dai, Andrew M.; Józefowicz, Rafał; Bengio, Samy

doi:10.48550/arxiv.1511.06349

Cited by 167 publications

(292 citation statements)

References 0 publications

Supporting

Mentioning

290

Contrasting

Order By: Relevance

“…Training VRAG involves optimizing two objectives -reducing the KL-divergence between the document-prior and document-posterior, and, maximizing the log likelihood of the responses. VAE models often end up prioritizing the KL-divergence over the likelihood objective and sometimes end up with zero KLdivergence by forcing the document-posterior to match the prior (called posterior-collapse) (Lucas et al 2019;Bowman et al 2015;Chen et al 2016;Oord, Vinyals, and Kavukcuoglu 2017). However, we hypothesize even in cases where there is no posterior collapse, the joint training could result in the response-generator (likelihood term) being inadequately trained.…”

Section: Effect Of Decoder Fine-tuningmentioning

confidence: 89%

“…These models are trained end-to-end by maximizing the likelihood of responses on a given training set of conversational logs (Sordoni et al 2015). However real-world conversational systems also need to be able to incorporate external (structured or unstructured) knowledge while generating responses (Gangi Reddy et al 2019;Budzianowski et al 2018;Qu et al 2020;. Existing work for knowledge grounded tasks typically relies on the availability of conversation logs and the associated knowledge source instance (Qu et al 2020;Gangi Reddy et al 2019), though some recent methods relax the dependence on annotated logs (Raghu, Gupta et al 2020;Lewis et al 2020;Guu et al 2020).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Variational Learning for Unsupervised Knowledge Grounded Dialogs

Mishra,

Madan,

Pandey

et al. 2021

Preprint

View full text Add to dashboard Cite

Recent methods for knowledge grounded dialogs generate responses by incorporating information from an external textual document (Lewis et al. 2020;Guu et al. 2020). These methods do not require the exact document to be known during training and rely on the use of a retrieval system to fetch relevant documents from a large index. The documents used to generate the responses are modeled as latent variables whose prior probabilities need to be estimated. Models such as RAG (Lewis et al. 2020), marginalize the document probabilities over the documents retrieved from the index to define the log likelihood loss function which is optimized end-to-end. In this paper, we develop a variational approach to the above technique wherein, we instead maximize the Evidence Lower bound (ELBO). Using a collection of three publicly available open-conversation datasets, we demonstrate how the posterior distribution, that has information from the ground-truth response, allows for a better approximation of the objective function during training. To overcome the challenges associated with sampling over a large knowledge collection, we develop an efficient approach to approximate the ELBO. To the best of our knowledge we are the first to apply variational training for open-scale unsupervised knowledge grounded dialog systems.1 typical k = 5-10. Note that this an approximation of the document-prior distribution -sampling a document based on the prior distribution, would require retrieval scores for each document in the index collection which can be expensive to compute (Guu et al. 2020;Lewis et al. 2020)

show abstract

Section: Effect Of Decoder Fine-tuningmentioning

confidence: 89%

Section: Introductionmentioning

confidence: 99%

Variational Learning for Unsupervised Knowledge Grounded Dialogs

Mishra,

Madan,

Pandey

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…By contrast, we aim to generate a sequence from a latent space whose dimension is invariant of sequence length. In this respect, the proposed method is similar to RNNbased variational autoencoders (VAE) [26], [27]. However, the assumption that the latent distribution is Gaussian is not always realistic and restricts the generation performance because realworld data may follow a much more complex distribution.…”

Section: Related Workmentioning

confidence: 99%

Towards Generating Real-World Time Series Data

Pei¹,

Ren²,

Yang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Time series data generation has drawn increasing attention in recent years. Several generative adversarial network (GAN) based methods have been proposed to tackle the problem usually with the assumption that the targeted time series data are well-formatted and complete. However, real-world time series (RTS) data are far away from this utopia, e.g., long sequences with variable lengths and informative missing data raise intractable challenges for designing powerful generation algorithms. In this paper, we propose a novel generative framework for RTS data -RTSGAN to tackle the aforementioned challenges. RTSGAN first learns an encoder-decoder module which provides a mapping between a time series instance and a fixeddimension latent vector and then learns a generation module to generate vectors in the same latent space. By combining the generator and the decoder, RTSGAN is able to generate RTS which respect the original feature distributions and the temporal dynamics. To generate time series with missing values, we further equip RTSGAN with an observation embedding layer and a decide-and-generate decoder to better utilize the informative missing patterns. Experiments on the four RTS datasets show that the proposed framework outperforms the previous generation methods in terms of synthetic data utility for downstream classification and prediction tasks. Our code is available at https://seqml.github.io/rtsgan.

show abstract

“…Another category of representation learning methods is VAE and its variants [22], [23]. Bowman et al introduce an RNN-based VAE to model the latent properties of sentences [24], which inspires us to learn the traits of drivers from their trajectories. Conditional VAEs (CVAE) are widely used in pedestrian and vehicle trajectory predictions since discrete latent states can represent different behavior modes such as braking and turning [25]- [28].…”

Section: B Representation Learning For Sequential Datamentioning

confidence: 99%

Learning to Navigate Intersections with Unsupervised Driver Trait Inference

Liu¹,

Chang²,

Chen³

et al. 2021

Preprint

View full text Add to dashboard Cite

Navigation through uncontrolled intersections is one of the key challenges for autonomous vehicles. Identifying the subtle differences in hidden traits of other drivers can bring significant benefits when navigating in such environments. We propose an unsupervised method for inferring driver traits such as driving styles from observed vehicle trajectories. We use a variational autoencoder with recurrent neural networks to learn a latent representation of traits without any ground truth trait labels. Then, we use this trait representation to learn a policy for an autonomous vehicle to navigate through a T-intersection with deep reinforcement learning. Our pipeline enables the autonomous vehicle to adjust its actions when dealing with drivers of different traits to ensure safety and efficiency. Our method demonstrates promising performance and outperforms state-of-the-art baselines in the T-intersection scenario.

show abstract

Generating Sentences from a Continuous Space

Cited by 167 publications

References 0 publications

Variational Learning for Unsupervised Knowledge Grounded Dialogs

Variational Learning for Unsupervised Knowledge Grounded Dialogs

Towards Generating Real-World Time Series Data

Learning to Navigate Intersections with Unsupervised Driver Trait Inference

Contact Info

Product

Resources

About