Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoders

Liu, Fangyu; Vulić, Ivan; Korhonen, Anna; Collier, Nigel

doi:10.18653/v1/2021.emnlp-main.109

Cited by 51 publications

(49 citation statements)

References 54 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While SEMB and AOC variants exhibit similar performance, ISO variants perform much worse. The direct comparison between ISO and AOC demonstrates the importance of contextual information and seemingly limited usability of off-the-shelf multilingual encoders as word encoders, if no context is available, and if they are not further specialized to encode word-level information (Liu et al 2021). Similarity-specialized multilingual encoders, which rely on pretraining with parallel data, yield mixed results.…”

Section: Document-level Clir Resultsmentioning

confidence: 99%

On cross-lingual retrieval with multilingual text encoders

et al. 2022

Self Cite

View full text Add to dashboard Cite

Pretrained multilingual text encoders based on neural transformer architectures, such as multilingual BERT (mBERT) and XLM, have recently become a default paradigm for cross-lingual transfer of natural language processing models, rendering cross-lingual word embedding spaces (CLWEs) effectively obsolete. In this work we present a systematic empirical study focused on the suitability of the state-of-the-art multilingual encoders for cross-lingual document and sentence retrieval tasks across a number of diverse language pairs. We first treat these models as multilingual text encoders and benchmark their performance in unsupervised ad-hoc sentence- and document-level CLIR. In contrast to supervised language understanding, our results indicate that for unsupervised document-level CLIR—a setup with no relevance judgments for IR-specific fine-tuning—pretrained multilingual encoders on average fail to significantly outperform earlier models based on CLWEs. For sentence-level retrieval, we do obtain state-of-the-art performance: the peak scores, however, are met by multilingual encoders that have been further specialized, in a supervised fashion, for sentence understanding tasks, rather than using their vanilla ‘off-the-shelf’ variants. Following these results, we introduce localized relevance matching for document-level CLIR, where we independently score a query against document sections. In the second part, we evaluate multilingual encoders fine-tuned in a supervised fashion (i.e., we learn to rank) on English relevance data in a series of zero-shot language and domain transfer CLIR experiments. Our results show that, despite the supervision, and due to the domain and language shift, supervised re-ranking rarely improves the performance of multilingual transformers as unsupervised base rankers. Finally, only with in-domain contrastive fine-tuning (i.e., same domain, only language transfer), we manage to improve the ranking quality. We uncover substantial empirical differences between cross-lingual retrieval results and results of (zero-shot) cross-lingual transfer for monolingual retrieval in target languages, which point to “monolingual overfitting” of retrieval models trained on monolingual (English) data, even if they are based on multilingual transformers.

show abstract

Section: Document-level Clir Resultsmentioning

confidence: 99%

On cross-lingual retrieval with multilingual text encoders

et al. 2022

Self Cite

View full text Add to dashboard Cite

show abstract

“…Inference Configuration. For similarity-based inference, as in previous work (Liu et al, 2021a) the Mirror-BERT procedure relies on the 10k most frequent English words for contrastive learning. 5 For constrained beam search, used with the LP task, we set the hyperparameter K to 50.…”

Section: Methodsmentioning

confidence: 99%

“…Based on prior findings concerning multilingual PLMs (Liu et al, 2021b) and our own preliminary experiments, out-of-the-box Prix-LM produces entity embeddings of insufficient quality. However, we can transform them into entity encoders via a simple and efficient unsupervised Mirror-BERT procedure (Liu et al, 2021a). In short, Mirror-BERT is a contrastive learning method that calibrates PLMs and converts them into strong universal lexical or sentence encoders.…”

Section: Inferencementioning

confidence: 99%

“…First, we compare to two representative and widely used KG embedding models: 8 (i) TransE (Bordes et al, 2013) interprets relations as translations from source to target entities, while (ii) RotatE interprets relations 5 We use English words only for simplicity and direct comparisons. According to Liu et al (2021a), Mirror-BERT tuning which uses words from the actual test language pair might yield even better performance. Our training configurations are identical to the original Mirror-BERT paper, except for that we use a smaller batch size (128 instead of 200) due to hardware constraints.…”

Section: Link Predictionmentioning

confidence: 99%

See 1 more Smart Citation

Prix-LM: Pretraining for Multilingual Knowledge Base Construction

Zhou¹,

Liu²,

Vulić³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Knowledge bases (KBs) contain plenty of structured world and commonsense knowledge. As such, they often complement distributional text-based information and facilitate various downstream tasks. Since their manual construction is resource-and timeintensive, recent efforts have tried leveraging large pretrained language models (PLMs) to generate additional monolingual knowledge facts for KBs. However, such methods have not been attempted for building and enriching multilingual KBs. Besides wider application, such multilingual KBs can provide richer combined knowledge than monolingual (e.g., English) KBs. Knowledge expressed in different languages may be complementary and unequally distributed: this implies that the knowledge available in high-resource languages can be transferred to low-resource ones.To achieve this, it is crucial to represent multilingual knowledge in a shared/unified space. To this end, we propose a unified framework, Prix-LM , for multilingual KB construction and completion. We leverage two types of knowledge, monolingual triples and crosslingual links, extracted from existing multilingual KBs, and tune a multilingual language encoder XLM-R via a causal language modeling objective. Prix-LM integrates useful multilingual and KB-based factual knowledge into a single model. Experiments on standard entityrelated tasks, such as link prediction in multiple languages, cross-lingual entity linking and bilingual lexicon induction, demonstrate its effectiveness, with gains reported over strong task-specialised baselines. 1 * Indicating equal contribution. 1 We release our code and model at: https://github. com/luka-group/prix-lm.

show abstract

“…Unlike prior work which conducted large-scale conversational pretraining from scratch using large datasets, we demonstrate that full pretraining is not needed to obtain universal conversational encoders. By leveraging the general semantic knowledge already stored in pretrained LMs, we can expose (i.e., 'rewire') that knowledge Gao et al, 2021b;Liu et al, 2021b) via much cheaper and quicker adaptive fine-tuning on a tiny fraction of the full Reddit data (e.g., even using < 0.01% of the Reddit corpus). Further, the task-oriented S2 CONVFIT-ing transforms pretrained LMs into task-specialised sentence encoders.…”

Section: Introductionmentioning

confidence: 99%

ConvFiT: Conversational Fine-Tuning of Pretrained Language Models

Vulić¹,

Su²,

Coope³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Transformer-based language models (LMs) pretrained on large text collections are proven to store a wealth of semantic knowledge. However, 1) they are not effective as sentence encoders when used off-the-shelf, and 2) thus typically lag behind conversationally pretrained (e.g., via response selection) encoders on conversational tasks such as intent detection (ID). In this work, we propose CON-VFIT, a simple and efficient two-stage procedure which turns any pretrained LM into a universal conversational encoder (after Stage 1 CONVFIT-ing) and task-specialised sentence encoder (after Stage 2). We demonstrate that 1) full-blown conversational pretraining is not required, and that LMs can be quickly transformed into effective conversational encoders with much smaller amounts of unannotated data; 2) pretrained LMs can be fine-tuned into task-specialised sentence encoders, optimised for the fine-grained semantics of a particular task. Consequently, such specialised sentence encoders allow for treating ID as a simple semantic similarity task based on interpretable nearest neighbours retrieval. We validate the robustness and versatility of the CON-VFIT framework with such similarity-based inference on the standard ID evaluation sets: CONVFIT-ed LMs achieve state-of-the-art ID performance across the board, with particular gains in the most challenging, few-shot setups.

show abstract

Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoders

Cited by 51 publications

References 54 publications

On cross-lingual retrieval with multilingual text encoders

On cross-lingual retrieval with multilingual text encoders

Prix-LM: Pretraining for Multilingual Knowledge Base Construction

ConvFiT: Conversational Fine-Tuning of Pretrained Language Models

Contact Info

Product

Resources

About