Better Word Embeddings by Disentangling Contextual n-Gram Information

Abstract

Pre-trained word vectors are ubiquitous in Natural Language Processing applications. In this paper, we show how training word embeddings jointly with bigram and even trigram embeddings, results in improved unigram embeddings. We claim that training word embeddings along with higher n-gram embeddings helps in the removal of the contextual information from the unigrams, resulting in better stand-alone word embeddings. We empirically show the validity of our hypothesis by outperforming other competing word representation models by a significant margin on a wide variety of tasks. We make our models publicly available.

Semantic search with domain-specific word-embedding and production monitoring in Fintech

“…From the processed data, uni-, bi-, and trigrams are jointly trained in deriving our custom word embeddings. Previous work suggests that multigram can improve on the quality of the obtained word embeddings (Gupta et al, 2019). In addition to word embeddings, we also derive document embeddings based on the cleaned text using pre-trained contextual multilingual universal sentence encoders.…”

Learning distributed sentence vectors with bi-directional 3D convolutions

“…Research works have been carried out to model the order of words when learning the distributed sentence representation (Le and Mikolov, 2014;Kiros et al, 2015;Conneau et al, 2017;Pagliardini et al, 2018;Gupta et al, 2019;Shen et al, 2019). Le and Mikolov propose Doc2vec (Le and Mikolov, 2014) to add a paragraph vector to represent the missing information from the current context.…”

“…Sent2Vec (Pagliardini et al, 2018) aims to strike a balance between matrix factorization and deep learning. Gupta et al (2019) propose two modifications of Word2vec by considering higher-order word n-grams along with uni-gram during training. Shen et al (2019) use InferSent (Conneau et al, 2017) for sentence embeddings based on word vectors learned by Glove (Pennington et al, 2014) or FastText (Joulin et al, 2017).…”

“…Shen et al (2019) use InferSent (Conneau et al, 2017) for sentence embeddings based on word vectors learned by Glove (Pennington et al, 2014) or FastText (Joulin et al, 2017). Gupta et al (2019) claim that training word embeddings along with higher n-gram embeddings helps in the removal of the contextual information from the uni-gram, resulting in better stand-alone word embeddings. All the aforementioned methods require pre-trained word vectors as input.…”

“…Although FastSent is faster than Skip-Thought in training, it sacrifices the order of words in a sentence, which is important in language models, such as the n-gram feature. For example, Gupta et al (2019) utilize the bi-gram and even tri-gram to train their embedding model.…”