Wikipedia Entities as Rendezvous across Languages: Grounding Multilingual Language Models by Predicting Wikipedia Hyperlinks

Calixto, Iacer; Raganato, Alessandro; Pasini, Tommaso

doi:10.18653/v1/2021.naacl-main.286

Cited by 9 publications

(11 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our experiments have demonstrated that entity supervision in EASE improves the quality of sentence embeddings both in the monolingual setting and, in particular, the multilingual setting. As recent studies have shown, entity annotations can be used as anchors to learn quality cross-lingual representations (Calixto et al, 2021;Nishikawa et al, 2021;Jian et al, 2022;Ri et al, 2022), and our work is another demonstration of their utility, particularly in sentence embeddings. One promising future direction is exploring how to better exploit the cross-lingual nature of entities.…”

Section: Discussionmentioning

confidence: 65%

See 1 more Smart Citation

EASE: Entity-Aware Contrastive Learning of Sentence Embedding

Nishikawa¹,

Ri²,

Yamada³

et al. 2022

Preprint

View full text Add to dashboard Cite

We present EASE, a novel method for learning sentence embeddings via contrastive learning between sentences and their related entities. The advantage of using entity supervision is twofold: (1) entities have been shown to be a strong indicator of text semantics and thus should provide rich training signals for sentence embeddings; (2) entities are defined independently of languages and thus offer useful cross-lingual alignment supervision. We evaluate EASE against other unsupervised models both in monolingual and multilingual settings. We show that EASE exhibits competitive or better performance in English semantic textual similarity (STS) and short text clustering (STC) tasks and it significantly outperforms baseline methods in multilingual settings on a variety of tasks. Our source code, pretrained models, and newly constructed multilingual STC dataset are available at https: //github.com/studio-ousia/ease.

show abstract

Section: Discussionmentioning

confidence: 65%

“…thus offer a useful cross-lingual alignment supervision (Calixto et al, 2021;Nishikawa et al, 2021;Jian et al, 2022;Ri et al, 2022). The extensive multilingual support of Wikipedia alleviates the need for a parallel resource to train well-aligned multilingual sentence embeddings, especially for low-resource languages.…”

Section: Introductionmentioning

confidence: 99%

EASE: Entity-Aware Contrastive Learning of Sentence Embedding

Nishikawa¹,

Ri²,

Yamada³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…While these works use hyperlinks to learn retrievers, we focus on using hyperlinks to create better context for learning general-purpose LMs. Separately, Calixto et al (2021) use Wikipedia hyperlinks to learn multilingual LMs. Citation links are often used to improve summarization and recommendation of academic papers (Qazvinian and Radev, 2008;Yasunaga et al, 2019;Bhagavatula et al, 2018;Khadka et al, 2020;Cohan et al, 2020).…”

Section: Related Workmentioning

confidence: 99%

LinkBERT: Pretraining Language Models with Document Links

Yasunaga¹,

Leskovec²,

Liang³

2022

Preprint

View full text Add to dashboard Cite

Language model (LM) pretraining can learn various knowledge from text corpora, helping downstream tasks. However, existing methods such as BERT model a single document, and do not capture dependencies or knowledge that span across documents. In this work, we propose LinkBERT, an LM pretraining method that leverages links between documents, e.g., hyperlinks. Given a text corpus, we view it as a graph of documents and create LM inputs by placing linked documents in the same context. We then pretrain the LM with two joint self-supervised objectives: masked language modeling and our new proposal, document relation prediction. We show that LinkBERT outperforms BERT on various downstream tasks across two domains: the general domain (pretrained on Wikipedia with hyperlinks) and biomedical domain (pretrained on PubMed with citation links). LinkBERT is especially effective for multi-hop reasoning and few-shot QA (+5% absolute improvement on HotpotQA and TriviaQA), and our biomedical LinkBERT sets new states of the art on various BioNLP tasks (+7% on BioASQ and USMLE). We release our pretrained models, LinkBERT and BioLinkBERT, as well as code and data. 1 * Equal senior authorship. 1 Available at https://github.com/michiyasunaga/ LinkBERT.

show abstract

“…initialization) of Prix-LM. 15 With the notable exception of Calixto et al (2021) who rely on the prediction of Wikipedia hyperlinks as an auxiliary/intermediate task to improve XLM-R's multilingual representation space for cross-lingual transfer, there has not been any work on augmenting multilingual PLMs with structured knowledge. Previous work has indicated that off-the-shelf mBERT and XLM-R fail on knowledge-intensive multilingual NLP tasks such as entity linking and KG completion, and especially so for low-resource languages (Liu et al, 2021b).…”

Section: Related Workmentioning

confidence: 99%

Prix-LM: Pretraining for Multilingual Knowledge Base Construction

Zhou¹,

Liu²,

Vulić³

et al. 2021

Preprint

View full text Add to dashboard Cite

Knowledge bases (KBs) contain plenty of structured world and commonsense knowledge. As such, they often complement distributional text-based information and facilitate various downstream tasks. Since their manual construction is resource-and timeintensive, recent efforts have tried leveraging large pretrained language models (PLMs) to generate additional monolingual knowledge facts for KBs. However, such methods have not been attempted for building and enriching multilingual KBs. Besides wider application, such multilingual KBs can provide richer combined knowledge than monolingual (e.g., English) KBs. Knowledge expressed in different languages may be complementary and unequally distributed: this implies that the knowledge available in high-resource languages can be transferred to low-resource ones.To achieve this, it is crucial to represent multilingual knowledge in a shared/unified space. To this end, we propose a unified framework, Prix-LM , for multilingual KB construction and completion. We leverage two types of knowledge, monolingual triples and crosslingual links, extracted from existing multilingual KBs, and tune a multilingual language encoder XLM-R via a causal language modeling objective. Prix-LM integrates useful multilingual and KB-based factual knowledge into a single model. Experiments on standard entityrelated tasks, such as link prediction in multiple languages, cross-lingual entity linking and bilingual lexicon induction, demonstrate its effectiveness, with gains reported over strong task-specialised baselines. 1 * Indicating equal contribution. 1 We release our code and model at: https://github. com/luka-group/prix-lm.

show abstract

Wikipedia Entities as Rendezvous across Languages: Grounding Multilingual Language Models by Predicting Wikipedia Hyperlinks

Cited by 9 publications

References 33 publications

EASE: Entity-Aware Contrastive Learning of Sentence Embedding

EASE: Entity-Aware Contrastive Learning of Sentence Embedding

LinkBERT: Pretraining Language Models with Document Links

Prix-LM: Pretraining for Multilingual Knowledge Base Construction

Contact Info

Product

Resources

About