K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters

Abstract

We study the problem of injecting knowledge into large pre-trained models like BERT and RoBERTa. Existing methods typically update the original parameters of pre-trained models when injecting knowledge. However, when multiple kinds of knowledge are injected, they may suffer from catastrophic forgetting. To address this, we propose K-ADAPTER, which remains the original parameters of the pre-trained model fixed and supports continual knowledge infusion. Taking RoBERTa as the pre-trained model, K-ADAPTER has a neural adapter for each kind of infused knowledge, like a plug-in connected to RoBERTa. There is no information flow between different adapters, thus different adapters are efficiently trained in a distributed way. We inject two kinds of knowledge, including factual knowledge obtained from automatically aligned text-triplets on Wikipedia and Wikidata, and linguistic knowledge obtained from dependency parsing. Results on three knowledge-driven tasks (total six datasets) including relation classification, entity typing and question answering demonstrate that each adapter improves the performance, and the combination of both adapters brings further improvements. Probing experiments further show that K-ADAPTER captures richer factual and commonsense knowledge than RoBERTa.

Knowledge Enhanced Pretrained Language Models: A Compreshensive Survey

“…These encyclopedia KG are able to provide abundant knowledge for PLMs to integrate. A majority of exiting work on KE-PLMs [108][26][105] [96][90] [82][66] uses Wikidata 1 as knowledge source. Typically, entities in Wikidata are linked with entity mentions in the text of Wikipedia.…”

“…Though most existing approaches exploit only one knowledge source, it is worth noting that certain methods attempt to incorporate from more than one knowledge source. For example, K-Adapter [96] incorporate knowledge from multiple sources by learning a different adapter for each knowledge source. It exploits both dependency relation as linguistic knowledge and relation/fact knowledge from Wikidata.…”

“…ERICA [66] applies contrastive loss on entity and relation, which pulls neighbor entity/relation close and pushes non-neighbors far apart in the embedding space. K-Adapter [96] introduces a factual adapter which incorporates relation information by performing relation classification based on the entity context. K-BERT [49] injects knowledge by augmenting sentence with the triplets from KG to transform it into a knowledge-rich sentence tree.…”

“…Since many existing approaches incorporate entity-level knowledge, entity related tasks (e.g., entity typing and relation classification) become natural testbeds for evaluating the efficacy of these KE-PLMs. By injecting entity information Knowledge Source ERICA [66] Yes entity/relation discrimination Wikipedia, Wikidata ERNIE (THU) [108] Yes entity prediction Wikipedia/Wikidata ERNIE 2.0 (Baidu) [83] Yes masked entity/phrase N/A E-BERT [64] Yes entity/wordpiece alignment Wikipedia2Vec E(commerce)-BERT [106] Yes neighbor Product Reconstruction product graph/AutoPhrase[75] EaE [21] Yes mention detection/linking Wikipedia CokeBERT [80] Yes entity prediction Wikipedia/Wikidata COMET [6] No autoregressive ATOMIC, ConceptNet K-Adapter [96] No dependency relation Wikipedia, Wikidata, Stanford Parser KnowBERT [61] Yes entity linking WordNet, Wikipedia K-BERT [49] No finetuning WikiZh, WebtextZh, CN-DBpedia HowNet, MedicalKG KEPLER [97] Yes TransE scoring Wikipedia/Wikidata KG-BERT [102] Yes relation cross-entropy ConceptNet KG-BART [51] Yes masked concept ConceptNet KgPLM [26] Yes generative/discriminative masked entity Wikipedia/Wikidata FaE [90] Yes masked entity et.al Wikipedia/Wikidata JAKET [105] Yes entity category/relation type/masked entity Wikipedia/Wikidata LUKE [100] Yes entity prediction Wikipedia WKLM [99] Yes entity replacement detection Wikipedia/Wikidata CoLAKE [82] Yes masked entity prediction Wikipedia/Wikidata KT-NET [101] No finetuning N/A LIBERT [40] Yes lexical relation prediction WordNet SenseBERT [41] Yes supersense prediction WordNet Syntax-BERT [2] No masks induced by syntax tree parsing syntax tree SentiLARE [36] Yes POS/ word level polarity/sentiment polarity SentiWordNet [44] No finetuning ConceptNet COCOLM [104] Yes discourse relation/co-occurrence relation ASER [22] Yes autoregressive ConceptNet/ATOMIC AMS [103] Yes distractor-based loss ConceptNet GLM …”

“…These observations motivate work on designing more knowledge-aware pre-trained models. Recently, an evergrowing body of work aims at explicitly incorporating knowledge into PLMs [100] [108] [61][90] [96][49] [33]. They exploit knowledge from various sources such as encyclopedia knowledge, commonsense knowledge and linguistic knowledge with different injection strategies.…”