Combining Static and Contextualised Multilingual Embeddings

Abstract

Static and contextual multilingual embeddings have complementary strengths. Static embeddings, while less expressive than contextual language models, can be more straightforwardly aligned across multiple languages. We combine the strengths of static and contextual models to improve multilingual representations. We extract static embeddings for 40 languages from XLM-R, validate those embeddings with cross-lingual word retrieval, and then align them using VecMap. This results in high-quality, highly multilingual static embeddings. Then we apply a novel continued pre-training approach to XLM-R, leveraging the high quality alignment of our static embeddings to better align the representation space of XLM-R. We show positive results for multiple complex semantic tasks. We release the static embeddings and the continued pretraining code. 1 Unlike most previous work, our continued pre-training approach does not require parallel text.

Biomedical Knowledge Discovery From Unstructured Text Corpora Using Contextual Word Embeddings

“…Importantly, the static-equivalent embeddings produced from contextualized embeddings can be utilized in identical ways as those from older Word2Vec or GloVe models, and also outperform them [9]. Subsequently, novel methods of creating static-equivalents have been described, using continuous bag-of-word approaches [23], phrases [24] and by combining contextual and static embeddings [25], for example. Nevertheless, this study has demonstrated as the n A Proposed Knowledge Discovery Method Utilizing Contextual Word Embeddings Based upon the results of this study, a working knowledge-discovery framework utilizing BERT can be achieved by:…”