On Commonsense Cues in BERT for Solving Commonsense Tasks

Cui, Leyang; Cheng, Sijie; Wu, Yu; Zhang, Yue

doi:10.18653/v1/2021.findings-acl.61

Cited by 9 publications

(11 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There is a wide variety of datasets constructed to facilitate the development of novel approaches to the problem of NLI (Storks et al, 2019). The task has evolved within a series of RTE challenges (Dagan et al, 2005) and now comprises several standardized benchmark datasets such as SICK (Marelli et al, 2014), SNLI (Bowman et al, 2015), MNLI (Williams et al, 2018), and XNLI (Conneau et al, 2018b). Despite the rapid progress, recent work has found that these benchmarks may contain biases and annotation artifacts which raise questions whether state-of-the-art models indeed have or acquire the inference abilities (Tsuchiya 2018;Belinkov et al, 2019).…”

Section: Related Work Nli and Diagnostic Datasetsmentioning

confidence: 99%

Ad astra or astray: Exploring linguistic knowledge of multilingual BERT through NLI task

et al. 2022

View full text Add to dashboard Cite

Recent research has reported that standard fine-tuning approaches can be unstable due to being prone to various sources of randomness, including but not limited to weight initialization, training data order, and hardware. Such brittleness can lead to different evaluation results, prediction confidences, and generalization inconsistency of the same models independently fine-tuned under the same experimental setup. Our paper explores this problem in natural language inference, a common task in benchmarking practices, and extends the ongoing research to the multilingual setting. We propose six novel textual entailment and broad-coverage diagnostic datasets for French, German, and Swedish. Our key findings are that the mBERT model demonstrates fine-tuning instability for categories that involve lexical semantics, logic, and predicate-argument structure and struggles to learn monotonicity, negation, numeracy, and symmetry. We also observe that using extra training data only in English can enhance the generalization performance and fine-tuning stability, which we attribute to the cross-lingual transfer capabilities. However, the ratio of particular features in the additional training data might rather hurt the performance for model instances. We are publicly releasing the datasets, hoping to foster the diagnostic investigation of language models (LMs) in a cross-lingual scenario, particularly in terms of benchmarking, which might promote a more holistic understanding of multilingualism in LMs and cross-lingual knowledge transfer.

show abstract

Section: Related Work Nli and Diagnostic Datasetsmentioning

confidence: 99%

Ad astra or astray: Exploring linguistic knowledge of multilingual BERT through NLI task

et al. 2022

View full text Add to dashboard Cite

show abstract

“…Commonsense reasoning is the ability to reason about the underlying nature of situations humans encounter on a day-to-day basis such as the effects of Newtonian physics and the intentions of others. LMs are shown to possess a certain amount of commonsense knowledge in their parameters (Petroni et al, 2019;Davison et al, 2019;Cui et al, 2021). As a result, Huang et al (2019); Talmor et al (2019); Sap et al (2019); West et al (2021) introduce datasets to evaluate the extent to which LMs can reason over the knowledge they learned during pretraining.…”

Section: Commonsense Reasoningmentioning

confidence: 99%

A Review on Language Models as Knowledge Bases

Badr¹,

Li²,

Çelikyılmaz³

et al. 2022

Preprint

View full text Add to dashboard Cite

Recently, there has been a surge of interest in the NLP community on the use of pretrained Language Models (LMs) as Knowledge Bases (KBs). Researchers have shown that LMs trained on a sufficiently large (web) corpus will encode a significant amount of knowledge implicitly in its parameters. The resulting LM can be probed for different kinds of knowledge and thus acting as a KB. This has a major advantage over traditional KBs in that this method requires no human supervision. In this paper, we present a set of aspects that we deem an LM should have to fully act as a KB, and review the recent literature with respect to those aspects. 1

show abstract

“…In the past few years, large pre-trained language models (PLMs) have achieved state-of-the-art performance on many natural language processing tasks (Devlin et al, 2018;Liu et al, 2019b). Recent studies suggest that PLMs have possessed various kinds of knowledge into contextual representations (Goldberg, 2019;Petroni et al, 2019;Lin et al, 2019;Cui et al, 2021). However, the ability of PLMs to interpret similes remains under-explored.…”

Section: Introductionmentioning

confidence: 99%

Can Pre-trained Language Models Interpret Similes as Smart as Human?

He¹,

Cheng²,

Li³

et al. 2022

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

Self Cite

View full text Add to dashboard Cite

Simile interpretation is a crucial task in natural language processing. Nowadays, pre-trained language models (PLMs) have achieved stateof-the-art performance on many tasks. However, it remains under-explored whether PLMs can interpret similes or not. In this paper, we investigate the ability of PLMs in simile interpretation by designing a novel task named Simile Property Probing, i.e., to let the PLMs infer the shared properties of similes. We construct our simile property probing datasets from both general textual corpora and humandesigned questions, containing 1,633 examples covering seven main categories. Our empirical study based on the constructed datasets shows that PLMs can infer similes' shared properties while still underperforming humans. To bridge the gap with human performance, we additionally design a knowledge-enhanced training objective by incorporating the simile knowledge into PLMs via knowledge embedding methods. Our method results in a gain of 8.58% in the probing task and 1.37% in the downstream task of sentiment classification. The datasets and code are publicly available at https://github.com/Abbey4799/PLMs-Interpret-Simile.

show abstract

On Commonsense Cues in BERT for Solving Commonsense Tasks

Cited by 9 publications

References 32 publications

Ad astra or astray: Exploring linguistic knowledge of multilingual BERT through NLI task

Ad astra or astray: Exploring linguistic knowledge of multilingual BERT through NLI task

A Review on Language Models as Knowledge Bases

Can Pre-trained Language Models Interpret Similes as Smart as Human?

Contact Info

Product

Resources

About