Generated Knowledge Prompting for Commonsense Reasoning

Liu, Jiacheng; Liu, Alisa; Lu, Ximing; Welleck, Sean; West, Peter; Bras, Ronan Le; Choi, Yejin; Hajishirzi, Hannaneh

doi:10.18653/v1/2022.acl-long.225

Cited by 86 publications

(71 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…For example, when the final label is binary, Jung et al (2022) induce a tree of explanations, then use an SAT solver and an NLI verifier to infer the satisfiability of each explanation. For commonsense reasoning tasks, Liu et al (2022a) generate relevant knowledge as additional inputs to the model, to improve the performance. Another line of work proposes to better retrieve prompts closer to the target question to further improve task performance (Liu et al, 2022b;Rubin et al, 2021).…”

Section: Related Workmentioning

confidence: 99%

Rationale-Augmented Ensembles in Language Models

Wang¹,

Lee²,

Schuurmans³

et al. 2022

Preprint

View full text Add to dashboard Cite

Recent research has shown that rationales, or step-by-step chains of thought, can be used to improve performance in multi-step reasoning tasks. We reconsider rationale-augmented prompting for few-shot in-context learning, where (input → output) prompts are expanded to (input, rationale → output) prompts. For rationale-augmented prompting we demonstrate how existing approaches, which rely on manual prompt engineering, are subject to sub-optimal rationales that may harm performance. To mitigate this brittleness, we propose a unified framework of rationale-augmented ensembles, where we identify rationale sampling in the output space as the key component to robustly improve performance. This framework is general and can easily be extended to common natural language processing tasks, even those that do not traditionally leverage intermediate steps, such as question answering, word sense disambiguation, and sentiment analysis. We demonstrate that rationale-augmented ensembles achieve more accurate and interpretable results than existing prompting approaches-including standard prompting without rationales and rationale-based chain-of-thought prompting-while simultaneously improving interpretability of model predictions through the associated rationales.Preprint.

show abstract

Section: Related Workmentioning

confidence: 99%

Rationale-Augmented Ensembles in Language Models

Wang¹,

Lee²,

Schuurmans³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…They utilize a multilingual pretrained XLM-RoBERTabase [3] model along with prompting methods to conduct natural language understanding tasks in 15 languages. Furthermore, most recent research for prompt-based learning has been applied in various fields of natural language processing including relation extraction [22], commonsense reasoning [8], and complementing weakness of prompt [4,15,23]. Son et al [22] introduced a multitask learning approach for predicting a relation in a dialogue by guiding the model on the relational cues with an MLM-based relational mention prediction and the prior distribution of entity types.…”

Section: Related Workmentioning

confidence: 99%

“…Son et al [22] introduced a multitask learning approach for predicting a relation in a dialogue by guiding the model on the relational cues with an MLM-based relational mention prediction and the prior distribution of entity types. Liu et al [8] proposed generated knowledge prompting to obtain the external knowledge required to solve commonsense reasoning tasks. Cui et al [4] proposed a soft prototype verbalizer to find a suitable verbalizer within a large vocab.…”

Section: Related Workmentioning

confidence: 99%

Plain Template Insertion: Korean-Prompt-Based Engineering for Few-Shot Learners

et al. 2022

View full text Add to dashboard Cite

Prompt-based learning is a method used for language models to interpret natural language by remembering the prior knowledge acquired and the training objective. Recent prompt-based fewshot learners have achieved superior performance by alleviating the catastrophic forgetting that occurs in pretrained language models. Few-shot learning contributes towards solving the data scarcity problem, an enormous challenge in AI systems and a significant consideration in natural language processing research. In spite of the significance of few-shot learning, research on Korean language-based few-shot learning is insufficient, and whether the prompt-based approach is appropriate for the Korean language has not been thoroughly verified. As a step toward realizing a Korean-prompt-based few-shot learner, we attempt to apply prompt engineering to the Korean language understanding benchmark dataset and introduce plain template insertion to overcome data scarcity in a more practical few-shot setting. The contributions of this study are as follows: (1) presumably, this is the first study to apply prompt-based few-shot learning to Korean benchmark datasets. With 32 few-shot settings, it improves performance by +14.88, +29.04, and +1.81 in the natural language inference, semantic textual similarity, and topic classification tasks. (2) We present prompt engineering, which merely inserts a plain template and increases data efficiency without training example selection, augmentation, reformulation, and retrieval. (3) Our approach is robust to the Korean prompt's contextual information and sentence structure and is applicable to both hard-and soft-prompt.

show abstract

“…Recent research has demonstrated that relevant knowledge can provide useful context for approaching commonsense tasks. Yet these methods either retrieve from indomain knowledge bases (Mitra et al, 2019;Chang et al, 2020) that do not have good coverage over commonsense, or generate knowledge from neural models (Shwartz et al, 2020;Gu et al, 2022;Liu et al, 2022), which often need domain-specific engineering and very large models (e.g. GPT-3 (Brown et al, 2020)).…”

Section: Introductionmentioning

confidence: 99%

Rainier: Reinforced Knowledge Introspector for Commonsense Question Answering

Liu¹,

Skyler²,

Lu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Knowledge underpins reasoning. Recent research demonstrates that when relevant knowledge is provided as additional context to commonsense question answering (QA), it can substantially enhance the performance even on top of state-of-the-art. The fundamental challenge is where and how to find such knowledge that is high quality and on point with respect to the question; knowledge retrieved from knowledge bases are incomplete and knowledge generated from language models are inconsistent.We present RAINIER 1 , or Reinforced Knowledge Introspector, that learns to generate contextually relevant knowledge in response to given questions. Our approach starts by imitating knowledge generated by GPT-3, then learns to generate its own knowledge via reinforcement learning where rewards are shaped based on the increased performance on the resulting question answering. RAINIER demonstrates substantial and consistent performance gains when tested over 9 different commonsense benchmarks: including 5 datasets that are seen during model training, as well as 4 datasets that are kept unseen. Our work is the first to report that knowledge generated by models that are orders of magnitude smaller than GPT-3, even without direct supervision on the knowledge itself, can exceed the quality of commonsense knowledge elicited from GPT-3.

show abstract

Generated Knowledge Prompting for Commonsense Reasoning

Cited by 86 publications

References 24 publications

Rationale-Augmented Ensembles in Language Models

Rationale-Augmented Ensembles in Language Models

Plain Template Insertion: Korean-Prompt-Based Engineering for Few-Shot Learners

Rainier: Reinforced Knowledge Introspector for Commonsense Question Answering

Contact Info

Product

Resources

About