Informative Language Encoding by Variational Autoencoders Using Transformer

Abstract

In natural language processing (NLP), Transformer is widely used and has reached the state-of-the-art level in numerous NLP tasks such as language modeling, summarization, and classification. Moreover, a variational autoencoder (VAE) is an efficient generative model in representation learning, combining deep learning with statistical inference in encoded representations. However, the use of VAE in natural language processing often brings forth practical difficulties such as a posterior collapse, also known as Kullback–Leibler (KL) vanishing. To mitigate this problem, while taking advantage of the parallelization of language data processing, we propose a new language representation model as the integration of two seemingly different deep learning models, which is a Transformer model solely coupled with a variational autoencoder. We compare the proposed model with previous works, such as a VAE connected with a recurrent neural network (RNN). Our experiments with four real-life datasets show that implementation with KL annealing mitigates posterior collapses. The results also show that the proposed Transformer model outperforms RNN-based models in reconstruction and representation learning, and that the encoded representations of the proposed model are more informative than other tested models.

TLN-LSTM: an automatic modulation recognition classifier based on a two-layer nested structure of LSTM network for extremely long signal sequences

“…However, when the length of the input time series is too long, the LSTM network will also encounter problems such as long training time, slow parameter update and even gradient disappearance (Hochreiter, 1991; Dandwate et al , 2023). At this time, LSTM needs to spend more time training a long sequence, and when using the backpropagation mechanism to update model parameters, a longer sequence will also cause the gradient to gradually decrease to close to zero, and the parameters cannot be updated, thereby losing the key information in the previous time step and the information in front of the long sequence, that is, long-term dependency problems cannot be dealt with (Ok et al , 2022; Mercan et al , 2023). For large LSTM models, a reasonable input sequence length is between 100 and 500 (Xiao et al , 2019).…”

A Tree-Transformer based VAE with fragment tokenization for large chemical models

“…Biesner et al [44] integrate the token vectors of each word from a Transformer encoder using an RNN to obtain a single latent variable. On the other hand, Ok et al [45] encodes the token vector of the sentence representation into a latent variable through a simple S5.…”

A Tree-Transformer based VAE with fragment tokenization for large chemical models

“…Biesner et al [41] integrate the token vectors of each word from a Transformer encoder using an RNN to obtain a single latent variable. On the other hand, Ok et al [42] encodes the token vector of the sentence representation into a latent variable through a simple linear layer. Since sequential processing with RNNs undermines the advantage of parallel processing offered by the Transformer, our model is based on the approach of Ok et al to construct the VAE model.…”