Learning Contextualized Knowledge Structures for Commonsense Reasoning

Yan, Jun; Raman, Mrigank; Chan, Aaron; Zhang, Tianyu; Rossi, Ryan A.; Zhao, Handong; Kim, Sung‐Chul; Lipka, Nedim; Ren, Xiang

doi:10.48550/arxiv.2010.12873

Cited by 5 publications

(2 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To study the effect of using KGs as external knowledge sources, we compare our method with vanilla fine-tuned LMs, which are knowledge-agnostic. We fine-tune RoBERTa- RGCN (Schlichtkrull et al, 2018) 72.7 (±0.2) 68.4 (±0.7) GconAttn 72.6 (±0.4) 68.6 (±1.0) KagNet (Lin et al, 2019) 73.5 (±0.2) 69.0 (±0.8) RN (Santoro et al, 2017) 74.6 (±0.9) 69.1 (±0.2) MHGRN (Feng et al, 2020) 74.5 (±0.1) 71.1 (±0.8) QA-GNN (Yasunaga et al, 2021) 76.5 (±0.2) 73.4 (±0.9) GREASELM (Ours) 78.5 (±0.5) 74.2 (±0.4) (Yan et al, 2020) 81.4 ≥355M AMR-SG (Xu et al, 2021) 81.6 ∼361M ALBERT + KPG (Wang et al, 2020) 81.8 ≥235M QA-GNN (Yasunaga et al, 2021) 82.8 ∼360M T5 * (Raffel et al, 2020) 83.2 ∼3B T5 + KB (Pirtoaca) 85.4 ≥11B UnifiedQA * (Khashabi et al, 2020) 87.2 ∼11B GREASELM (Ours) 84.8 ∼359M…”

Section: Baseline Methodsmentioning

confidence: 99%

GreaseLM: Graph REASoning Enhanced Language Models for Question Answering

Zhang¹,

Bosselut²,

Yasunaga³

et al. 2022

Preprint

View full text Add to dashboard Cite

Answering complex questions about textual narratives requires reasoning over both stated context and the world knowledge that underlies it. However, pretrained language models (LM), the foundation of most modern QA systems, do not robustly represent latent relationships between concepts, which is necessary for reasoning. While knowledge graphs (KG) are often used to augment LMs with structured representations of world knowledge, it remains an open question how to effectively fuse and reason over the KG representations and the language context, which provides situational constraints and nuances. In this work, we propose GREASELM, a new model that fuses encoded representations from pretrained LMs and graph neural networks over multiple layers of modality interaction operations. Information from both modalities propagates to the other, allowing language context representations to be grounded by structured world knowledge, and allowing linguistic nuances (e.g., negation, hedging) in the context to inform the graph representations of knowledge. Our results on three benchmarks in the commonsense reasoning (i.e., CommonsenseQA, OpenbookQA) and medical question answering (i.e., MedQA-USMLE) domains demonstrate that GREASELM can more reliably answer questions that require reasoning over both situational constraints and structured knowledge, even outperforming models 8× larger. 1

show abstract

Section: Baseline Methodsmentioning

confidence: 99%

GreaseLM: Graph REASoning Enhanced Language Models for Question Answering

Zhang¹,

Bosselut²,

Yasunaga³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…From Refs. [10,11], we know that such an approach can help improve commonsense reasoning performance. Considering Concept-Net knowledge about the problem, the model can read a dialogue with commonsense.…”

Section: Introductionmentioning

confidence: 99%

Considering Commonsense in Solving QA: Reading Comprehension with Semantic Search and Continual Learning

Park

Lim

2022

Applied Sciences

View full text Add to dashboard Cite

Unlike previous dialogue-based question-answering (QA) datasets, DREAM, multiple-choice Dialogue-based REAding comprehension exaMination dataset, requires a deep understanding of dialogue. Many problems require multi-sentence reasoning, whereas some require commonsense reasoning. However, most pre-trained language models (PTLMs) do not consider commonsense. In addition, because the maximum number of tokens that a language model (LM) can deal with is limited, the entire dialogue history cannot be included. The resulting information loss has an adverse effect on performance. To address these problems, we propose a Dialogue-based QA model with Common-sense Reasoning (DQACR), a language model that exploits Semantic Search and continual learning. We used Semantic Search to complement information loss from truncated dialogue. In addition, we used Semantic Search and continual learning to improve the PTLM’s commonsense reasoning. Our model achieves an improvement of approximately 1.5% over the baseline method and can thus facilitate QA-related tasks. It contributes toward not only dialogue-based QA tasks but also another form of QA datasets for future tasks.

show abstract

RiddleSense: Reasoning about Riddle Questions Featuring Linguistic Creativity and Commonsense Knowledge

Lin¹,

Wu²,

Yang³

et al. 2021

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021

View full text Add to dashboard Cite

Question: I have five fingers but I am not alive. What am I? Answer: a glove.Answering such a riddle-style question is a challenging cognitive process, in that it requires complex commonsense reasoning abilities, an understanding of figurative language, and counterfactual reasoning skills, which are all important abilities for advanced natural language understanding (NLU). However, there is currently no dataset aiming to test these abilities. In this paper, we present RIDDLE-SENSE 1 , a new multiple-choice question answering task, which comes with the first large dataset (5.7k examples) for answering riddlestyle commonsense questions. We systematically evaluate a wide range of models over the RIDDLESENSE challenge, and point out that there is a large gap between the bestsupervised model and human performancesuggesting intriguing future research in the direction of higher-order commonsense reasoning and linguistic creativity towards building advanced NLU systems.

show abstract

Learning Contextualized Knowledge Structures for Commonsense Reasoning

Cited by 5 publications

References 39 publications

GreaseLM: Graph REASoning Enhanced Language Models for Question Answering

GreaseLM: Graph REASoning Enhanced Language Models for Question Answering

Considering Commonsense in Solving QA: Reading Comprehension with Semantic Search and Continual Learning

RiddleSense: Reasoning about Riddle Questions Featuring Linguistic Creativity and Commonsense Knowledge

Contact Info

Product

Resources

About