UnCommonSense: Informative Negative Knowledge about Everyday Concepts

Arnaout, Hiba; Razniewski, Simon; Weikum, Gerhard; Pan, Jeff Z.

doi:10.1145/3511808.3557484

Cited by 7 publications

(5 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They propose a two-task assessment in which LLMs need to i) answer yes or no to world knowledge questions and ii) generate commonsense compelling sentences from related keywords. Some recent research has been directed to building knowledge bases in which negative commonsense is stored (Arnaout et al, 2022), in order to be reused for commonsense reasoning.…”

Section: Related Workmentioning

confidence: 99%

This is not a Dataset: A Large Negation Benchmark to Challenge Large Language Models

García-Ferrero,

Altuna,

Alvez

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Although large language models (LLMs) have apparently acquired a certain level of grammatical knowledge and the ability to make generalizations, they fail to interpret negation, a crucial step in Natural Language Processing. We try to clarify the reasons for the sub-optimal performance of LLMs understanding negation. We introduce a large semi-automatically generated dataset of circa 400,000 descriptive sentences about commonsense knowledge that can be true or false in which negation is present in about 2/3 of the corpus in different forms. We have used our dataset with the largest available open LLMs in a zero-shot approach to grasp their generalization and inference capability and we have also fine-tuned some of the models to assess whether the understanding of negation can be trained. Our findings show that, while LLMs are proficient at classifying affirmative sentences, they struggle with negative sentences and lack a deep understanding of negation, often relying on superficial cues. Although finetuning the models on negative sentences improves their performance, the lack of generalization in handling negation is persistent, highlighting the ongoing challenges of LLMs regarding negation understanding and generalization. The dataset and code are publicly available: https://github.com/hitz-zentroa/ This-is-not-a-Dataset

show abstract

Section: Related Workmentioning

confidence: 99%

This is not a Dataset: A Large Negation Benchmark to Challenge Large Language Models

García-Ferrero,

Altuna,

Alvez

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

show abstract

“…A new research area has emerged in the last few years, suggesting the importance of the explicit materialization of important negative statements about real-world subjects. Since then, several methodologies have been proposed [1,2,5,12,13]. The goal is to compile lists of statements (biographic summaries) about subjects, where the statements are truly negative, but also salient, unexpected, or normally mistaken as true positives.…”

Section: Introductionmentioning

confidence: 99%

“…To compile these lists, different data sources and methodologies have been explored. In [1,2], candidate salient negatives are derived from existing positive statements about highly related entities. The computation relies on the local closed-world assumption, an assumption of completeness over identified relevant subgraphs, coupled with ranking metrics such as relative frequencies.…”

Section: Introductionmentioning

confidence: 99%

“…Paris is..", and receive the completion the capital of france. A few papers focused on the ability of these models to store and understand negative knowledge [2,3,6]. In [6], using masked probing, authors found that LMs, such as BERT, struggle to understand negation, predicting fly for the probe "Birds cannot [MASK]".…”

Section: Introductionmentioning

confidence: 99%

“…In [6], using masked probing, authors found that LMs, such as BERT, struggle to understand negation, predicting fly for the probe "Birds cannot [MASK]". In [2], methods to infer negative statements from knowledge graphs and text have been compared on a more specific negation task, namely generating salient negative commonsense statements. Results of these models are compared to ones using GPT-3.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Negative Statements Considered Useful

et al. 2021

View full text Add to dashboard Cite

An Experimental Pipeline for Automated Reasoning in Natural Language (Short Paper)

Tammet,

Järv,

Verrev

et al. 2023

Lecture Notes in Computer Science

View full text Add to dashboard Cite

We describe an experimental implementation of a logic-based end-to-end pipeline of performing inference and giving explained answers to questions posed in natural language. The main components of the pipeline are semantic parsing, integration with large knowledge bases, automated reasoning using extended first order logic, and finally the translation of proofs back to natural language. While able to answer relatively simple questions on its own, the implementation is targeting research into building hybrid neurosymbolic systems for gaining trustworthiness and explainability. The end goal is to combine machine learning and large language models with the components of the implementation and to use the automated reasoner as an interface between natural language and external tools like database systems and scientific calculations.

show abstract

UnCommonSense: Informative Negative Knowledge about Everyday Concepts

Cited by 7 publications

References 20 publications

This is not a Dataset: A Large Negation Benchmark to Challenge Large Language Models

This is not a Dataset: A Large Negation Benchmark to Challenge Large Language Models

Negative Statements Considered Useful

An Experimental Pipeline for Automated Reasoning in Natural Language (Short Paper)

Contact Info

Product

Resources

About