Great~Truths~are ~Always ~Simple: A Rather Simple Knowledge Encoder for Enhancing the Commonsense Reasoning Capacity of Pre-Trained Models A Rather Simple Knowledge Encoder for Enhancing the Commonsense Reasoning Capacity of Pre-Trained Models

Abstract

Commonsense reasoning in natural language is a desired ability of artificial intelligent systems. For solving complex commonsense reasoning tasks, a typical solution is to enhance pre-trained language models (PTMs) with a knowledge-aware graph neural network (GNN) encoder that models a commonsense knowledge graph (CSKG). Despite the effectiveness, these approaches are built on heavy architectures, and can't clearly explain how external knowledge resources improve the reasoning capacity of PTMs. Considering this issue, we conduct a deep empirical analysis, and find that it is indeed relation features from CSKGs (but not node features) that mainly contribute to the performance improvement of PTMs. Based on this finding, we design a simple MLP-based knowledge encoder that utilizes statistical relation paths as features. Extensive experiments conducted on five benchmarks demonstrate the effectiveness of our approach, which also largely reduces the parameters for encoding CSKGs. Our codes and data are publicly available at https://github.com/RUCAIBox/SAFE.

Graph Reasoning Transformers for Knowledge-Aware Question Answering

“…GNNs are commonly used to model external KGs for conducting joint reasoning with LMs. Some works (Feng et al 2020;Yasunaga et al 2021;Jiang et al 2022) use the information of one modality to augment another modality. The most representative work, the QA-GNN (Yasunaga et al 2021), adds language representation as a new node to the retrieved KG and employs an elaborate GNN to jointly update the LM and GNN representations via message passing.…”

“…However, knowledge-aware QA is more concerned with identifying which knowledge triplets can support the answer, while the node information can be replaced by serial numbers to distinguish between different entities (Galárraga et al 2013). As a result, the initial node embeddings are demonstrated to be dispensable, and some GNN layers are shown to be overparameterized (Jiang et al 2022;Wang et al 2022). Different from the existing node-based models, we use knowledge triplets as atomic knowledge for graph encoding and propose a triplet encoder for graph encoding.…”

“…GconAttn (Wang et al 2019) 72.6 (±0.3) 68.5 (±0.9) RN (Santoro et al 2017) 74.5 (±0.9) 69.0 (±0.2) MHGRN (Feng et al 2020) 74.4 (±0.1) 71.1 (±0.8) QA-GNN (Yasunaga et al 2021) 76.5 (±0.2) 73.4 (±0.9) SAFE (Jiang et al 2022) -74.0 GreaseLM (Zhang et al 2022) 78.5 (±0.5) 74.2 (±0.4) GSC (Wang et al 2022) 79.1 (±0.2) 74.4 (±0.4) QAT (Park et al 2023) 79.5 (±0.4) 75.4 (±0.3) FIT (Ye et al 2023) 78.5 (±0.5) 75.6 (±0.3) GRT (ours) 79.6(±0.3) 76.1(±0.4)…”

“…While GNNs have been widely used to handle graph structures in downstream tasks such as node classification, node clustering, and link prediction, recent works have shown that GNNs are unable to fully utilize external knowledge graphs in knowledge-aware QA tasks. Jiang et al (2022) surprisingly found that even when reducing the dimensionality of the GNN node embeddings from 1024 to 1, the performance of KG-augmented QA systems was still unchanged. Wang et al (2022) leveraged SpareVD to detect which ingredients of a GNN are crucial for knowledge augmentation and found that even by ablating the node embedding and message passing methods, the performance of KG-augmented QA systems could surprisingly be improved.…”