Pragmatic competence of pre-trained language models through the lens of discourse connectives

Abstract

As pre-trained language models (LMs) continue to dominate NLP, it is increasingly important that we understand the depth of language capabilities in these models. In this paper, we target pre-trained LMs' competence in pragmatics, with a focus on pragmatics relating to discourse connectives. We formulate cloze-style tests using a combination of naturally-occurring data and controlled inputs drawn from psycholinguistics. We focus on testing models' ability to use pragmatic cues to predict discourse connectives, models' ability to understand implicatures relating to connectives, and the extent to which models show humanlike preferences regarding temporal dynamics of connectives. We find that although models predict connectives reasonably well in the context of naturally-occurring data, when we control contexts to isolate high-level pragmatic cues, model sensitivity is much lower. Models also do not show substantial humanlike temporal preferences. Overall, the findings suggest that at present, dominant pretraining paradigms do not result in substantial pragmatic competence in our models.

Towards Understanding Large-Scale Discourse Structures in Pre-Trained and Fine-Tuned Language Models

“…Further, Zhu et al (2020) use 24 hand-crafted rhetorical features to execute three different supervised probing tasks, showing promising performance of the BERT model. Similarly, Pandia et al (2021) aim to infer pragmatics through the prediction of discourse connectives by analyzing the model inputs and outputs and Koto et al (2021a) analyze discourse in PLMs through seven supervised probing tasks, finding that BART and BERT contain Looking at all these prior works analyzing the amount of discourse in PLMs, structures are solely explored through the use of proxy tasks, such as connective prediction (Pandia et al, 2021), relation classification (Kurfalı and Östling, 2021) and others (Koto et al, 2021a). However, despite the difficulties of encoding arbitrarily long documents, we believe that to systematically explore the relationship between PLMs and discourse, considering complete documents is imperative.…”

“…Besides their strong empirical results on most real-world problems, such as summarization (Zhang et al, 2020;Xiao et al, 2021a), question-answering (Joshi et al, 2020;Oguz et al, 2021) and sentiment analysis (Adhikari et al, 2019;, uncovering what kind of linguistic knowledge is captured by this new type of pre-trained language models (PLMs) has become a prominent question by itself. As part of this line of research, called BERTology (Rogers et al, 2020), researchers explore the amount of linguistic understanding encapsulated in PLMs, exposed through either external probing tasks (Raganato and Tiedemann, 2018;Zhu et al, 2020;Koto et al, 2021a) or unsupervised methods (Wu et al, 2020;Pandia et al, 2021) to analyze the syntactic structures (e.g., Hewitt and Manning (2019); Wu et al (2020)), relations (Papanikolaou et al, 2019), ontologies (Michael et al, 2020) and, to a more limited extend, discourse related behaviour (Zhu et al, 2020;Koto et al, 2021a;Pandia et al, 2021).…”

“…most information related to discourse. In contrast to the approach taken by both Zhu et al (2020) and Koto et al (2021a), we use an unsupervised methodology to test the amount of discourse information stored in PLMs (which can also conveniently be used to infer discourse structures for new and unseen documents) and extend the work by Pandia et al (2021) by taking a closer look at the internal workings of the self-attention component.Looking at all these prior works analyzing the amount of discourse in PLMs, structures are solely explored through the use of proxy tasks, such as connective prediction(Pandia et al, 2021), relation classification(Kurfalı and Östling, 2021) and others(Koto et al, 2021a). However, despite the difficulties of encoding arbitrarily long documents, we believe that to systematically explore the relationship between PLMs and discourse, considering complete documents is imperative.…”

Pre-trained Language Models’ Interpretation of Evaluativity Implicature: Evidence from Gradable Adjectives Usage in Context

“…A lot of attention has been paid to increase LMs' general transparency (Ettinger, 2020;Rogers et al, 2020), among which studies on LMs' interpretation of implicitness mostly focus on scalar implicature or presupposition (Schuster et al, 2020;Jeretic et al, 2020;Pandia et al, 2021). To our knowledge, no studies in this line have been done on gradable adjectives' EVAL implicature, although EVAL and gradability are classic topics in context sensitivity.…”