Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly

Kassner, Nora; Schütze, Hinrich

doi:10.18653/v1/2020.acl-main.698

Cited by 173 publications

(172 citation statements)

References 21 publications

Supporting

Mentioning

158

Contrasting

Order By: Relevance

“…Yet, for some reason the cook's eyes did not water". While the LM recognizes the lexical relatedness between onions and watering eyes, it is not sensitive to negation, as was recently shown for several other language models (Ettinger, 2020;Kassner and Schütze, 2020).…”

Section: Disconfirmed Expectationsmentioning

confidence: 70%

“…On the one hand, Petroni et al (2019) and Davison et al (2019) somewhat successfully used pre-trained LMs to complete commonsense KBs. On the other hand, Logan et al (2019) have shown that LMs are limited in their ability to generate accurate factual knowledge, and Kassner and Schütze (2020) and Ettinger (2020) pointed out that LMs are not sensitive to negation, resulting in generating incorrect facts ("birds can't fly"). Finally, Shwartz et al (2020b) showed that despite being noisy, knowledge generated by LMs can be used to improve performance on commonsense tasks.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Do Neural Language Models Overcome Reporting Bias?

Shwartz¹,

Choi²

2020

Proceedings of the 28th International Conference on Computational Linguistics

View full text Add to dashboard Cite

Mining commonsense knowledge from corpora suffers from reporting bias, over-representing the rare at the expense of the trivial (Gordon and Van Durme, 2013). We study to what extent pre-trained language models overcome this issue. We find that while their generalization capacity allows them to better estimate the plausibility of frequent but unspoken of actions, outcomes, and properties, they also tend to overestimate that of the very rare, amplifying the bias that already exists in their training corpus.

show abstract

Section: Disconfirmed Expectationsmentioning

confidence: 70%

Section: Related Workmentioning

confidence: 99%

Do Neural Language Models Overcome Reporting Bias?

Shwartz¹,

Choi²

2020

Proceedings of the 28th International Conference on Computational Linguistics

View full text Add to dashboard Cite

show abstract

“…While both works have shown somewhat promising results, other work showed that knowledge extracted from LMs is expectantly not always ac-curate. Specifically, Kassner and Schütze (2020) showed that negated facts are also considered likely by the LM, while Logan et al (2019) pointed out that LMs may over-generalize and produce incorrect facts such as "Barack Obama's wife is Hillary".…”

Section: Extracting Knowledge From Lmsmentioning

confidence: 99%

“…Despite the performance boost, LMs as knowledge providers suffer from various shortcomings: (i) insufficient coverage: due to reporting bias, many trivial facts might not be captured by LMs because they are rarely written about (Gordon and Van Durme, 2013). (ii) insufficient precision: the distributional training objective increases the probability of non-facts that are semantically similar to true facts, as in negation ("birds cannot fly"; Kassner and Schütze, 2020). LMs excel in predicting the semantic category of a missing word, but might predict the wrong instance in that category (e.g., depending on the phrasing, BERT sometimes predicts red as the color of a dove).…”

Section: Introductionmentioning

confidence: 99%

Unsupervised Commonsense Question Answering with Self-Talk

Shwartz¹,

West²,

Bras³

et al. 2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

132

160

View full text Add to dashboard Cite

Natural language understanding involves reading between the lines with implicit background knowledge. Current systems either rely on pretrained language models as the sole implicit source of world knowledge, or resort to external knowledge bases (KBs) to incorporate additional relevant knowledge. We propose an unsupervised framework based on self-talk as a novel alternative to multiple-choice commonsense tasks. Inspired by inquiry-based discovery learning (Bruner, 1961), our approach inquires language models with a number of information seeking questions such as "what is the definition of ..." to discover additional background knowledge. Empirical results demonstrate that the self-talk procedure substantially improves the performance of zeroshot language model baselines on four out of six commonsense benchmarks, and competes with models that obtain knowledge from external KBs. While our approach improves performance on several benchmarks, the selftalk induced knowledge even when leading to correct answers is not always seen as helpful by human judges, raising interesting questions about the inner-workings of pre-trained language models for commonsense reasoning.

show abstract

“…For example, our model predicts 'climate change' is an attributing factor in the comment 'there is no proof of climate change droughts and floods are all natural phenomenon they have happened before there were humans also' with high confidence. Our model fails to understand the negation as well as the context; perhaps due to a well-documented limitation of BERT's inability in handling negation (Kassner and Schütze, 2019).…”

Section: Error Analysismentioning

confidence: 97%

Social Media Attributions in the Context of Water Crisis

Sarkar

Mahinder²,

Sarkar

et al. 2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

View full text Add to dashboard Cite

Attribution of natural disasters/collective misfortune is a widely-studied political science problem. However, such studies typically rely on surveys, expert opinions, or external signals such as voting outcomes. In this paper, we explore the viability of using unstructured, noisy social media data to complement traditional surveys through automatically extracting attribution factors. We present a novel prediction task of attribution tie detection of identifying the factors (e.g., poor city planning, exploding population etc.) held responsible for the crisis in a social media document. We focus on the 2019 Chennai water crisis that rapidly escalated into a discussion topic with global importance following alarming water-crisis statistics. On a challenging data set constructed from YouTube comments (72,098 comments posted by 43,859 users on 623 videos relevant to the crisis), we present a neural baseline to identify attribution ties that achieves a reasonable performance (accuracy: 87.34% on attribution detection and 81.37% on attribution resolution). We release the first annotated data set of 2,500 comments in this important domain 1 .

show abstract

Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly

Cited by 173 publications

References 21 publications

Do Neural Language Models Overcome Reporting Bias?

Do Neural Language Models Overcome Reporting Bias?

Unsupervised Commonsense Question Answering with Self-Talk

Social Media Attributions in the Context of Water Crisis

Contact Info

Product

Resources

About