Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models

Kassner, Nora; Dufter, Philipp; Schütze, Hinrich

doi:10.18653/v1/2021.eacl-main.284

Cited by 54 publications

(62 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…"Dante was born in __X__". Follow up studies have introduced improved prompts for eliciting such knowledge (Jiang et al, 2020b) as well as multilingual versions (Jiang et al, 2020a;Kassner et al, 2021). However, all these benchmarks assume a static view of the knowledge inside an LM, and consider all answers across time to be correct for a given query.…”

Section: Related Workmentioning

confidence: 99%

Time-Aware Language Models as Temporal Knowledge Bases

Dhingra,

Cole,

Eisenschlos

et al. 2021

Preprint

View full text Add to dashboard Cite

Many facts come with an expiration date, from the name of the President to the basketball team Lebron James plays for. But language models (LMs) are trained on snapshots of data collected at a specific moment in time, and this can limit their utility, especially in the closed-book setting where the pretraining corpus must contain the facts the model should memorize. We introduce a diagnostic dataset aimed at probing LMs for factual knowledge that changes over time and highlight problems with LMs at either end of the spectrum-those trained on specific slices of temporal data, as well as those trained on a wide range of temporal data. To mitigate these problems, we propose a simple technique for jointly modeling text with its timestamp. This improves memorization of seen facts from the training time period, as well as calibration on predictions about unseen facts from future time periods. We also show that models trained with temporal context can be efficiently "refreshed" as new data arrives, without the need for retraining from scratch.

show abstract

Section: Related Workmentioning

confidence: 99%

Time-Aware Language Models as Temporal Knowledge Bases

Dhingra,

Cole,

Eisenschlos

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Knowledge Probing and Knowledge in BERT Our model's capability to memorize tracts of Wikipedia connects to past work on extracting knowledge from language models (Petroni et al, 2019); our task is easier because it is also possible to make type predictions based on context, giving our model a "backoff" capability. Recently, knowledge probing settings have been proposed for the multilingual setting as well (Kassner et al, 2021). Another related line of work attempts to add entity information into BERT (Zhang et al, 2019;Peters et al, 2019;Poerner et al, 2020;Févry et al, 2020, inter alia); our model could potentially benefit from this, but given our focus on generalizing to new entities, a model that more easily memorizes common entities may actually work less well.…”

Section: Related Workmentioning

confidence: 99%

Cross-Lingual Fine-Grained Entity Typing

Nila¹,

Onoe²,

Durrett³

2021

Preprint

View full text Add to dashboard Cite

The growth of cross-lingual pre-trained models has enabled NLP tools to rapidly generalize to new languages. While these models have been applied to tasks involving entities, their ability to explicitly predict typological features of these entities across languages has not been established. In this paper, we present a unified cross-lingual fine-grained entity typing model capable of handling over 100 languages and analyze this model's ability to generalize to languages and entities unseen during training. We train this model on cross-lingual training data collected from Wikipedia hyperlinks in multiple languages (training languages). During inference, our model takes an entity mention and context in a particular language (test language, possibly not in the training languages) and predicts fine-grained types for that entity. Generalizing to new languages and unseen entities are the fundamental challenges of this entity typing setup, so we focus our evaluation on these settings and compare against simple yet powerful string match baselines. Experimental results show that our approach outperforms the baselines on unseen languages such as Japanese, Tamil, Arabic, Serbian, and Persian. In addition, our approach substantially improves performance on unseen entities (even in unseen languages) over the baselines, and human evaluation shows a strong ability to predict relevant types in these settings.

show abstract

“…shown promising results in cross-lingual entitycentric tasks (Vulić et al, 2020;Liu et al, 2021b;Kassner et al, 2021). Note that the "Mirror-BERT" fine-tuning step is always applied, yielding an increase in performance.…”

Section: Models In Comparisonmentioning

confidence: 99%

“…Task Setup. Since we are interested in probing how much knowledge a PLM contains in multiple languages, we use the multilingual LAnguage Model Analysis (mLAMA) benchmark proposed by Kassner et al (2021). To ensure a strictly fair comparison, we only compare XLM-R and Prix-LM.…”

Section: Prompt-based Knowledge Probingmentioning

confidence: 99%

See 1 more Smart Citation

Prix-LM: Pretraining for Multilingual Knowledge Base Construction

Zhou¹,

Liu²,

Vulić³

et al. 2021

Preprint

View full text Add to dashboard Cite

Knowledge bases (KBs) contain plenty of structured world and commonsense knowledge. As such, they often complement distributional text-based information and facilitate various downstream tasks. Since their manual construction is resource-and timeintensive, recent efforts have tried leveraging large pretrained language models (PLMs) to generate additional monolingual knowledge facts for KBs. However, such methods have not been attempted for building and enriching multilingual KBs. Besides wider application, such multilingual KBs can provide richer combined knowledge than monolingual (e.g., English) KBs. Knowledge expressed in different languages may be complementary and unequally distributed: this implies that the knowledge available in high-resource languages can be transferred to low-resource ones.To achieve this, it is crucial to represent multilingual knowledge in a shared/unified space. To this end, we propose a unified framework, Prix-LM , for multilingual KB construction and completion. We leverage two types of knowledge, monolingual triples and crosslingual links, extracted from existing multilingual KBs, and tune a multilingual language encoder XLM-R via a causal language modeling objective. Prix-LM integrates useful multilingual and KB-based factual knowledge into a single model. Experiments on standard entityrelated tasks, such as link prediction in multiple languages, cross-lingual entity linking and bilingual lexicon induction, demonstrate its effectiveness, with gains reported over strong task-specialised baselines. 1 * Indicating equal contribution. 1 We release our code and model at: https://github. com/luka-group/prix-lm.

show abstract

Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models

Cited by 54 publications

References 18 publications

Time-Aware Language Models as Temporal Knowledge Bases

Time-Aware Language Models as Temporal Knowledge Bases

Cross-Lingual Fine-Grained Entity Typing

Prix-LM: Pretraining for Multilingual Knowledge Base Construction

Contact Info

Product

Resources

About