Exploiting Structured Knowledge in Text via Graph-Guided Representation Learning

Abstract

In this work, we aim at equipping pre-trained language models with structured knowledge. We present two self-supervised tasks learning over raw text with the guidance from knowledge graphs. Building upon entity-level masked language models, our first contribution is an entity masking scheme that exploits relational knowledge underlying the text. This is fulfilled by using a linked knowledge graph to select informative entities and then masking their mentions. In addition, we use knowledge graphs to obtain distractors for the masked entities, and propose a novel distractor-suppressed ranking objective that is optimized jointly with masked language model. In contrast to existing paradigms, our approach uses knowledge graphs implicitly, only during pre-training, to inject language models with structured knowledge via learning from raw text. It is more efficient than retrieval-based methods that perform entity linking and integration during finetuning and inference, and generalizes more effectively than the methods that directly learn from concatenated graph triples. Experiments show that our proposed model achieves improved performance on five benchmarks, including question answering and knowledge base completion. Method Dev Test BERT-large (Devlin et al., 2019) 66.0 64.5 RoBERTa-large (Liu et al., 2019b) 78.2 77.1 McQueen (RoBERTa) (Anonymous) 79.5 78.0 GB-KSI (Anonymous) 77.5 78.1 GLM (RoBERTa) 79.6 78.6Table 3: Results on SocialIQA. The results for comparative methods are copied from Sap et al. (2019b) or leaderboard.

Relational World Knowledge Representation in Contextual Language Models: A Review

“…What lexical cues or correlations should be allowed in knowledge probes? Progress in this direction will Masked token prediction Guu et al, 2020) Contrastive learning (Xiong et al, 2020;Shen et al, 2020) Text-to-KB links-late fusion ( § 4.2)…”

“…Contrastive learning techniques, which have been used for LM supervision at the word and sentence level , have also been devised for supervision on entity mentions (Shen et al, 2020). For example, Xiong et al ( 2020) replace a proportion of entity mentions in the pretraining corpus with the names of negatively-sampled entities of the same type, and train an LM to predict whether the entity in the span has been replaced (Figure 3b).…”

“…BERT has also been fine-tuned to match its output token representations to external entity embeddings for the task of end-to-end entity linking . Such approaches rely heavily on the quality of the externally-learned embeddings, which is both a strength and a drawback: Such embeddings may contain useful implicit structural information about the KB, but on the other hand may propagate errors into the LM (Shen et al, 2020).…”

NegatER: Unsupervised Discovery of Negatives in Commonsense Knowledge Bases

“…Bilinear AVG (Li et al, 2016) 91.70 DNN AVG (Li et al, 2016) 92.00 DNN LSTM (Li et al, 2016) 89.20 DNN AVG + CKBG (Saito et al, 2018) 94.70 Factorized (Jastrzębski et al, 2018) 79.40 Prototypical (Jastrzębski et al, 2018) 89.00 Concatenation (Davison et al, 2019) 68.80 Template (Davison et al, 2019) 72.20 Template + Grammar (Davison et al, 2019) 74.40 Coherency Ranking (Davison et al, 2019) 78.80 KG-BERT BERT-BASE (Shen et al, 2020) 93.20 KG-BERT GLM(RoBERTa-LARGE) (Shen et al, 2020) 94.60…”

Does Structure Matter? Encoding Documents for Machine Reading Comprehension

“…A series of recent work explores incorporating structured knowledge embedded in text into MRC (Shen et al, 2020;Dhingra et al, 2020). However, such kind of linking information for creating triples is not necessarily prominent in documents other than Wikipedia.…”