Contrastive Self-Supervised Learning for Commonsense Reasoning

Klein, Tassilo; Nabi, Moin

doi:10.18653/v1/2020.acl-main.671

Cited by 45 publications

(37 citation statements)

References 14 publications

(16 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Annotating more shots in the target language is an intuitive solution. Designing task-specific pretraining/finetuning objectives could also be promising (Klein and Nabi, 2020;Ram et al, 2021).…”

Section: Target-adapting Resultsmentioning

confidence: 99%

A Closer Look at Few-Shot Crosslingual Transfer: The Choice of Shots Matters

Zhao¹,

Zhu²,

Shareghi

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

Few-shot crosslingual transfer has been shown to outperform its zero-shot counterpart with pretrained encoders like multilingual BERT. Despite its growing popularity, little to no attention has been paid to standardizing and analyzing the design of few-shot experiments. In this work, we highlight a fundamental risk posed by this shortcoming, illustrating that the model exhibits a high degree of sensitivity to the selection of few shots. We conduct a largescale experimental study on 40 sets of sampled few shots for six diverse NLP tasks across up to 40 languages. We provide an analysis of success and failure cases of few-shot transfer, which highlights the role of lexical features. Additionally, we show that a straightforward full model finetuning approach is quite effective for few-shot transfer, outperforming several state-of-the-art few-shot approaches. As a step towards standardizing few-shot crosslingual experimental designs, we make our sampled few shots publicly available. 1 * Equal contribution.

show abstract

Section: Target-adapting Resultsmentioning

confidence: 99%

A Closer Look at Few-Shot Crosslingual Transfer: The Choice of Shots Matters

Zhao¹,

Zhu²,

Shareghi

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

show abstract

“…Knowledge-Enriched BERT: Incorporating external knowledge into BERT has been shown to be effective. Such external knowledge includes world (factual) knowledge for tasks such as entity typing and relation classification (Zhang et al, 2019;Peters et al, 2019;Liu et al, 2019a;Xiong et al, 2019), sentiment knowledge for sentiment analysis (Tian et al, 2020;, word sense knowledge for word sense disambiguation (Levine et al, 2019), commonsense knowledge for commonsense reasoning (Klein and Nabi, 2020) and sarcasm generation (Chakrabarty et al, 2020), legal knowledge for legal element extraction (Zhong et al, 2020), numerical skills for numerical reasoning (Geva et al, 2020), and coding knowledge for code generation . Biomedical BERT: BERT can also be enriched with biomedical knowledge via pre-training over biomedical corpora like PubMed, as in BioBERT (Lee et al, 2020), SciBERT (Beltagy et al, 2019), ClinicalBERT (Alsentzer et al, 2019) and Blue-BERT (Peng et al, 2019).…”

Section: Related Workmentioning

confidence: 99%

Infusing Disease Knowledge into BERT for Health Question Answering, Medical Inference and Disease Name Recognition

He¹,

Zhu²,

Zhang³

et al. 2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

View full text Add to dashboard Cite

Knowledge of a disease includes information of various aspects of the disease, such as signs and symptoms, diagnosis and treatment. This disease knowledge is critical for many healthrelated and biomedical tasks, including consumer health question answering, medical language inference and disease name recognition. While pre-trained language models like BERT have shown success in capturing syntactic, semantic, and world knowledge from text, we find they can be further complemented by specific information like knowledge of symptoms, diagnoses, treatments, and other disease aspects. Hence, we integrate BERT with disease knowledge for improving these important tasks. Specifically, we propose a new disease knowledge infusion training procedure and evaluate it on a suite of BERT models including BERT, BioBERT, SciBERT, Clinical-BERT, BlueBERT, and ALBERT. Experiments over the three tasks show that these models can be enhanced in nearly all cases, demonstrating the viability of disease knowledge infusion. For example, accuracy of BioBERT on consumer health question answering is improved from 68.29% to 72.09%, while new SOTA results are observed in two datasets. We make our data and code freely available.

show abstract

“…Trinh and Le (2018) use a pre-trained language model to score candidate sentences for the Pronoun Disambiguation and Winograd Schema Challenge (Levesque et al, 2012). Klein and Nabi (2020) use a sentence-level loss to enhance commonsense knowledge in BERT. Mao et al (2019) demonstrate that pre-trained language models fine-tuned on SWAG (Zellers et al, 2018) are able to provide commonsense grounding for story generation.…”

Section: Related Workmentioning

confidence: 99%

On Commonsense Cues in BERT for Solving Commonsense Tasks

Cui

Cheng

et al. 2021

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021

View full text Add to dashboard Cite

BERT has been used for solving commonsense tasks such as CommonsenseQA. While prior research has found that BERT does contain commonsense information to some extent, there has been work showing that pre-trained models can rely on spurious associations (e.g., data bias) rather than key cues in solving sentiment classification and other problems. We quantitatively investigate the presence of structural commonsense cues in BERT when solving commonsense tasks, and the importance of such cues for the model prediction. Using two different measures, we find that BERT does use relevant knowledge for solving the task, and the presence of commonsense knowledge is positively correlated to the model accuracy.

show abstract

Contrastive Self-Supervised Learning for Commonsense Reasoning

Cited by 45 publications

References 14 publications

A Closer Look at Few-Shot Crosslingual Transfer: The Choice of Shots Matters

A Closer Look at Few-Shot Crosslingual Transfer: The Choice of Shots Matters

Infusing Disease Knowledge into BERT for Health Question Answering, Medical Inference and Disease Name Recognition

On Commonsense Cues in BERT for Solving Commonsense Tasks

Contact Info

Product

Resources

About