A Bilingual Generative Transformer for Semantic Sentence Embedding

Abstract

Semantic sentence embedding models encode natural language sentences into vectors, such that closeness in embedding space indicates closeness in the semantics between the sentences. Bilingual data offers a useful signal for learning such embeddings: properties shared by both sentences in a translation pair are likely semantic, while divergent properties are likely stylistic or language-specific. We propose a deep latent variable model that attempts to perform source separation on parallel sentences, isolating what they have in common in a latent semantic vector, and explaining what is left over with language-specific latent vectors. Our proposed approach differs from past work on semantic sentence encoding in two ways. First, by using a variational probabilistic framework, we introduce priors that encourage source separation, and can use our model's posterior to predict sentence embeddings for monolingual data at test time. Second, we use high-capacity transformers as both data generating distributions and inference networkscontrasting with most past work on sentence embeddings. In experiments, our approach substantially outperforms the state-of-the-art on a standard suite of unsupervised semantic similarity evaluations. Further, we demonstrate that our approach yields the largest gains on more difficult subsets of these evaluations where simple word overlap is not a good indicator of similarity. 1

Self-training Improves Pre-training for Natural Language Understanding

“…In Table 10, we compare our sentence embedding method to previous approaches including BERT (Mean) (Devlin et al, 2018), InferSent (Conneau et al, 2017), GenSen (Subramanian et al, 2018), USE (Cer et al, 2018), Sentence-BERT (Reimers and Gurevych, 2019), uSIF (Ethayarajh, 2018a), Charagram (Wieting and Gimpel, 2017) and BGT (Wieting et al, 2019b). On average, our embeddings outperform previous approaches by 0.2% on STS 2012 to 2016 (Agirre et al, 2012(Agirre et al, , 2013(Agirre et al, , 2014(Agirre et al, , 2015(Agirre et al, , 2016, and by 0.9% on STS-Benchmark (Cer et al, 2017…”

Disentangling Semantics and Syntax in Sentence Embeddings with Pre-trained Language Models

“…Various sentence embedding models have been proposed in recent years. Most of these models utilize supervision from parallel data Artetxe and Schwenk, 2019b;Wieting et al, 2019Wieting et al, , 2020, natural language inference data (Conneau et al, 2017;Cer et al, 2018;Reimers and Gurevych, 2019), or a combination of both (Subramanian et al, 2018).…”

“…The constituency parse trees of all sentences are obtained from Stanford CoreNLP (Manning et al, 2014). We finetune a 6-layer BART base encoder as the semantic Wieting et al (2020). *BGT is evaluated on an additional dataset from STS13, which is not included in the standard SentEval toolkit.…”

“…Baselines We compare our model with other sentence embeddings models, including InferSent (Conneau et al, 2017), Universal Sentence Encoder (USE) (Cer et al, 2018), Sentence-BERT base (Reimers and Gurevych, 2019), VGVAE , and BGT (Wieting et al, 2020). We also include mean-pooled BERT base and BART base embeddings.…”

“…None of the previous models we compare in this work involves any hyperparameter search. The results for BGT are taken from Wieting et al (2020). For all other sentence embedding models, we use the trained model provided by their respective authors.…”