Rainier: Reinforced Knowledge Introspector for Commonsense Question Answering

Liu, Jiacheng; Skyler, Hallinan,; Lu, Ximing; He, Ping; Welleck, Sean; Hajishirzi, Hannaneh; Choi, Yejin

doi:10.48550/arxiv.2210.03078

Cited by 3 publications

(2 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Furthermore, our work can also be viewed from the perspective of learning discrete prompts for language models. Past work propose to generate knowledge pieces (Liu et al, 2022) or arbitrary textual snippets (Deng et al, 2022) which they append to the input via reinforcement learning. These works are different than ours in that their policy is conditioned solely on the input x whereas in our case we sample critiques of machine-generated predictions based on x and ŷ.…”

Section: Adapters and Discrete Prompt Learningmentioning

confidence: 99%

RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputs

Akyürek¹,

Ekin²,

Kalyan³

et al. 2023

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

View full text Add to dashboard Cite

Despite their unprecedented success, even the largest language models make mistakes. Similar to how humans learn and improve using feedback, previous work proposed providing language models with natural language feedback to guide them in repairing their outputs. Because human-generated critiques are expensive to obtain, researchers have devised learned critique generators in lieu of human critics while assuming one can train downstream models to utilize generated feedback. However, this approach does not apply to black-box or limited access models such as ChatGPT, as they cannot be fine-tuned. Moreover, in the era of large general-purpose language agents, fine-tuning is neither computationally nor spatially efficient as it results in multiple copies of the network. In this work, we introduce RL4F (Reinforcement Learning for Feedback), a multi-agent collaborative framework where the critique generator is trained to maximize end-task performance of GPT-3, a fixed model more than 200 times its size. RL4F produces critiques that help GPT-3 revise its outputs. We study three datasets for action planning, summarization and alphabetization and show relative improvements up to 10% in multiple text similarity metrics over other learned, retrievalaugmented or prompting-based critique generators. 1

show abstract

Section: Adapters and Discrete Prompt Learningmentioning

confidence: 99%

RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputs

Akyürek¹,

Ekin²,

Kalyan³

et al. 2023

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

View full text Add to dashboard Cite

show abstract

“…Question answering (QA) has become one of the most popular downstream tasks in natural language processing (NLP) in recent years. QA tasks utilize large-scale pre-trained language models (LMs) to obtain token representations, exemplified by BERT [1], GPT [2], ELMo [3], and RoBERTa [4], which have all achieved remarkable success. Meanwhile, commonsense as external knowledge is essential for QA systems to predict the correct answer, which is natural knowledge for humans [5].…”

Section: Introductionmentioning

confidence: 99%

Retrieval-Augmented Knowledge Graph Reasoning for Commonsense Question Answering

Sha

Feng

et al. 2023

Mathematics

View full text Add to dashboard Cite

Existing knowledge graph (KG) models for commonsense question answering present two challenges: (i) existing methods retrieve entities related to questions from the knowledge graph, which may extract noise and irrelevant nodes, and (ii) there is a lack of interaction representation between questions and graph entities. However, current methods mainly focus on retrieving relevant entities with some noisy and irrelevant nodes. In this paper, we propose a novel retrieval-augmented knowledge graph (RAKG) model, which solves the above issues using two key innovations. First, we leverage the density matrix to make the model reason along the corrected knowledge path and extract an enhanced subgraph of the knowledge graph. Second, we fuse representations of questions and graph entities through a bidirectional attention strategy, in which two representations fuse and update using a graph convolutional network (GCN). To evaluate the performance of our method, we conducted experiments on two widely used benchmark datasets: CommonsenseQA and OpenBookQA. The case study gives insight into the finding that the augmented subgraph provides reasoning along the corrected knowledge path for question answering.

show abstract

Retrieval-augmented Knowledge Graph Reasoning for Commonsense Question Answering

Sha¹,

Feng²,

He³

et al. 2023

Preprint

View full text Add to dashboard Cite

Existing Knowledge Graph (KG) models for commonsense question answering present two challenges: (i) existing methods retrieved entities related to questions from the knowledge graph, which may extract noise and irrelevant nodes, and (ii) lack of interaction representation between questions and graph entities. However, current methods mainly focus on retrieving relevant entities with some noisy and irrelevant nodes. In this paper, we propose a novel Retrieval-augmented Knowledge Graph (RAKG) model, which solves the above issues through two key innovations. First, we leverage the density matrix to make the model reason along the corrected knowledge path and extract an enhanced knowledge graph subgraph. Second, we fuse representations of questions and graph entities through a bidirectional attention strategy, in which two representations fuse and update by Graph Convolutional Network (GCN). To evaluate the performance of our method, we conduct experiments on two widely-used benchmark datasets CommonsenseQA and OpenBookQA. The case study gives insight into findings that the augmented subgraph provides reasoning along the corrected knowledge path for question answering.

show abstract

Rainier: Reinforced Knowledge Introspector for Commonsense Question Answering

Cited by 3 publications

References 27 publications

RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputs

RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputs

Retrieval-Augmented Knowledge Graph Reasoning for Commonsense Question Answering

Retrieval-augmented Knowledge Graph Reasoning for Commonsense Question Answering

Contact Info

Product

Resources

About