Benchmarking Commonsense Knowledge Base Population with an Effective Evaluation Dataset

Abstract

Reasoning over commonsense knowledge bases (CSKBs) whose elements are in the form of free-text is an important yet hard task in NLP. While CSKB completion only fills the missing links within the domain of the CSKB, CSKB population is alternatively proposed with the goal of reasoning unseen assertions from external resources. In this task, CSKBs are grounded to a large-scale eventuality (activity, state, and event) graph to discriminate whether novel triples from the eventuality graph are plausible or not. However, existing evaluations on the population task are either not accurate (automatic evaluation with randomly sampled negative examples) or of small scale (human annotation). In this paper, we benchmark the CSKB population task with a new large-scale dataset by first aligning four popular CSKBs, and then presenting a highquality human-annotated evaluation set to probe neural models' commonsense reasoning ability. We also propose a novel inductive commonsense reasoning model that reasons over graphs. Experimental results show that generalizing commonsense reasoning on unseen assertions is inherently a hard task. Models achieving high accuracy during training perform poorly on the evaluation set, with a large gap between human performance. Codes and data are available at https://github.com/ HKUST-KnowComp/CSKB-Population.

HKUST-Landslide Susceptibility Dataset (HKUST-LSD): A benchmark dataset for landslide susceptibility assessment in Hong Kong

“…The overall inter-annotator agreement (IAA) stands at 81% in terms of pairwise agreement, and the Fleiss kappa [20] is 0.56. These statistics are generally comparable to or slightly higher than those of other high-quality dataset construction works [54,18,19,29], which indicates that the annotators are close to achieving a strong internal agreement.…”

Gold: A Global and Local-aware Denoising Framework for Commonsense Knowledge Graph Noise Detection

“…While existing methods bank on expensive and timeconsuming crowdsourcing to collect commonsense knowledge (Sap et al, 2019a;Mostafazadeh et al, 2020), it remains infeasible to obtain CSKGs that are large enough to cover numerous entities and situations in the world (He et al, 2022;Tandon et al, 2014). To overcome this limitation, various automatic CSKG construction methods have been proposed to acquire commonsense knowledge at scale (Bosselut et al, 2019), including prompting Large Language Model (LLM) (West et al, 2022;Yu et al, 2022), rule mining from massive corpora (Tandon et al, 2017;Zhang et al, 2022a), and knowledge graph population (Fang et al, 2021a(Fang et al, ,b, 2023. Although those methods are effective, they still suffer from noises introduced by construction bias and the lack of human supervision.…”