Beyond Distributional Hypothesis: Let Language Models Learn Meaning-Text Correspondence

Abstract

The logical negation property (LNP), which implies generating different predictions for semantically opposite inputs (p is true iff ¬p is false), is an important property that a trustworthy language model must satisfy. However, much recent evidence shows that large-size pre-trained language models (PLMs) do not satisfy this property. In this paper, we perform experiments using probing tasks to assess PLMs' LNP understanding. Unlike previous studies that only examined negation expressions, we expand the boundary of the investigation to lexical semantics. Through experiments, we observe that PLMs violate the LNP frequently. To alleviate the issue, we propose a novel intermediate training task, named meaning-matching, designed to directly learn a meaning-text correspondence, instead of relying on the distributional hypothesis. Through multiple experiments, we find that the task enables PLMs to learn lexical semantic information. Also, through fine-tuning experiments on 7 GLUE tasks, we confirm that it is a safe intermediate task that guarantees a similar or better performance of downstream tasks. Finally, we observe that our proposed approach 1 outperforms our previous counterparts despite its time and resource efficiency.

ChatGPT Incorrectness Detection in Software Reviews

“…Similar to other LLMs, ChatGPT also suffers from a lack of consistency [32,33]. The model frequently alters its decisions when presented with a paraphrased sentence, revealing self-contradictory behavior.…”

Consistency Analysis of ChatGPT

“…To alleviate the issue, several works adopted data augmentation to train a model with abundant data containing negation expressions (Asai and Hajishirzi, 2020;Hosseini et al, 2021). Jang et al (2022b) introduced the meaning-matching task to enhance PLMs' textual understanding ability and observed performance improvements.…”

“…A correct behaviour is a crucial aspect in deciding models' trustworthiness by improving the certification 1 process (Jang et al, 2022a). In this regard, we mainly investigate the trustworthiness of ChatGPT in terms of logically consistent behaviour.…”

“…In this regard, we mainly investigate the trustworthiness of ChatGPT in terms of logically consistent behaviour. By using the BECEL dataset (Jang et al, 2022a), which is designed to ascertain whether language models satisfy various types of consistency, we analyse ChatGPT's ability to generate logically consistent predictions based on the four properties below:…”

Generalised Winograd Schema and its Contextuality

“…In previous work, we showed how natural language notions of context can be modelled by the mathematics of quantum contextuality [36,37,34]. In particular, we modelled anaphoric context in [28].…”