Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queries

Heinzerling, Benjamin; Inui, Kentaro

doi:10.18653/v1/2021.eacl-main.153

Cited by 51 publications

(28 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…On the other hand, there has been increasing interest in the use of large pretrained generative models as a source of knowledge. Heinzerling and Inui (2021) argue that large pretrained models can in fact serve as knowledge bases, while Davison et al (2019) argue that pretrained models can accurately assess the validity of knowledge mined from raw text. While pretrained models are also trained on raw texts, similar to knowledge bases and graphs, they draw from this knowledge probabilistically and thus are a distinct approach.…”

Section: External Knowledgementioning

confidence: 99%

Explaining Toxic Text via Knowledge Enhanced Text Generation

Sridhar¹,

Yang²

2022

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

Warning: This paper contains content that is offensive and may be upsetting.Biased or toxic speech can be harmful to various demographic groups. Therefore, it is not only important for models to detect these speech, but to also output explanations of why a given text is toxic. Previous literature has mostly focused on classifying and detecting toxic speech, and existing efforts on explaining stereotypes in toxic speech mainly use standard text generation approaches, resulting in generic and repetitive explanations. Building on these prior works, we introduce a novel knowledgeinformed encoder-decoder framework to utilize multiple knowledge sources to generate implications of biased text. Experiments show that our knowledge informed models outperform prior state-of-the-art models significantly, and can generate detailed explanations of stereotypes in toxic speech compared to baselines, both quantitatively and qualitatively.

show abstract

Section: External Knowledgementioning

confidence: 99%

Explaining Toxic Text via Knowledge Enhanced Text Generation

Sridhar¹,

Yang²

2022

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

show abstract

“…Tangential emerging research areas that are relevant to our work are knowledge acquisition via pre-trained language models and prompt-engineering. Prior work (such as [11], [16] or [48]) uses the knowledge within pre-trained language models for QA, fact checking or truthful generation. Significant efforts have focused on building better prompts and a representative collection can be found in a recent survey [19].…”

Section: Related Workmentioning

confidence: 99%

Rows from Many Sources: Enriching row completions from Wikidata with a pre-trained Language Model

Negreanu¹,

Karaoglu²,

Williams³

et al. 2022

Preprint

View full text Add to dashboard Cite

Row completion is the task of augmenting a given table of text and numbers with additional, relevant rows. The task divides into two steps: subject suggestion, the task of populating the main column; and gap filling, the task of populating the remaining columns. We present state-of-the-art results for subject suggestion and gap filling measured on a standard benchmark (WikiTables).Our idea is to solve this task by harmoniously combining knowledge base table interpretation and free text generation. We interpret the table using the knowledge base to suggest new rows and generate metadata like headers through property linking. To improve candidate diversity, we synthesize additional rows using free text generation via GPT-3, and crucially, we exploit the metadata we interpret to produce better prompts for text generation. Finally, we verify that the additional synthesized content can be linked to the knowledge base or a trusted web source such as Wikipedia.

show abstract

“…Prior efforts have also applied template and probe-based methods (Bouraoui et al, 2020;Petroni et al, 2019;Jiang et al, 2020b;Heinzerling and Inui, 2020) to extract relational knowledge from large pretrained models; we draw upon these techniques in this work. However, these works focus on general domain knowledge extraction, rather than clinical tasks which pose unique privacy concerns.…”

Section: Related Workmentioning

confidence: 99%

Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?

Lehman¹,

Jain²,

Pichotta³

et al. 2021

Preprint

View full text Add to dashboard Cite

Large Transformers pretrained over clinical notes from Electronic Health Records (EHR) have afforded substantial gains in performance on predictive clinical tasks. The cost of training such models (and the necessity of data access to do so) coupled with their utility motivates parameter sharing, i.e., the release of pretrained models such as ClinicalBERT (Alsentzer et al., 2019). While most efforts have used deidentified EHR, many researchers have access to large sets of sensitive, nondeidentified EHR with which they might train a BERT model (or similar). Would it be safe to release the weights of such a model if they did? In this work, we design a battery of approaches intended to recover Personal Health Information (PHI) from a trained BERT. Specifically, we attempt to recover patient names and conditions with which they are associated. We find that simple probing methods are not able to meaningfully extract sensitive information from BERT trained over the MIMIC-III corpus of EHR. However, more sophisticated "attacks" may succeed in doing so: To facilitate such research, we make our experimental setup and baseline probing models available. 1

show abstract

Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queries

Cited by 51 publications

References 43 publications

Explaining Toxic Text via Knowledge Enhanced Text Generation

Explaining Toxic Text via Knowledge Enhanced Text Generation

Rows from Many Sources: Enriching row completions from Wikidata with a pre-trained Language Model

Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?

Contact Info

Product

Resources

About