Examining the rhetorical capacities of neural language models

Abstract

Recently, neural language models (LMs) have demonstrated impressive abilities in generating high-quality discourse. While many recent papers have analyzed the syntactic aspects encoded in LMs, to date, there has been no analysis of the inter-sentential, rhetorical knowledge. In this paper, we propose a method that quantitatively evaluates the rhetorical capacities of neural LMs. We examine the capacities of neural LMs understanding the rhetoric of discourse by evaluating their abilities to encode a set of linguistic features derived from Rhetorical Structure Theory (RST). Our experiments show that BERT-based LMs outperform other Transformer LMs, revealing the richer discourse knowledge in their intermediate layer representations. In addition, GPT-2 and XLNet apparently encode less rhetorical knowledge, and we suggest an explanation drawing from linguistic philosophy. Our method presents an avenue towards quantifying the rhetorical capacities of neural LMs.

Towards Understanding Large-Scale Discourse Structures in Pre-Trained and Fine-Tuned Language Models

“…At the base of our work are two of the most popular and frequently used PLMs: BERT (Devlin et al, 2018) and BART (Lewis et al, 2020). We choose these two popular approaches in our study due to their complementary nature (encoder-only vs. encoder-decoder) and based on previous work by Zhu et al (2020) and Koto et al (2021a), showing the effectiveness of BERT and BART models for discourse related tasks.…”

“…Compared to our work, their tree inference approach is however computationally expensive and only explores the outputs of the BERT model. Further, Zhu et al (2020) use 24 hand-crafted rhetorical features to execute three different supervised probing tasks, showing promising performance of the BERT model. Similarly, Pandia et al (2021) aim to infer pragmatics through the prediction of discourse connectives by analyzing the model inputs and outputs and Koto et al (2021a) analyze discourse in PLMs through seven supervised probing tasks, finding that BART and BERT contain Looking at all these prior works analyzing the amount of discourse in PLMs, structures are solely explored through the use of proxy tasks, such as connective prediction (Pandia et al, 2021), relation classification (Kurfalı and Östling, 2021) and others (Koto et al, 2021a).…”

“…We select the BERT-base (110 million parameters) and BART-large (406 million parameters) models for our experiments. We choose these models for their diverse objectives (encoder-only vs. encoderdecoder), popularity for diverse fine-tuning tasks, and their prior exploration in regards to discourse (Zhu et al, 2020;Koto et al, 2021a). For the BARTlarge model, we limit our analysis to the encoder, as motivated in Koto et al (2021a), leaving experiments with the decoder for future work.…”

“…Besides their strong empirical results on most real-world problems, such as summarization (Zhang et al, 2020;Xiao et al, 2021a), question-answering (Joshi et al, 2020;Oguz et al, 2021) and sentiment analysis (Adhikari et al, 2019;, uncovering what kind of linguistic knowledge is captured by this new type of pre-trained language models (PLMs) has become a prominent question by itself. As part of this line of research, called BERTology (Rogers et al, 2020), researchers explore the amount of linguistic understanding encapsulated in PLMs, exposed through either external probing tasks (Raganato and Tiedemann, 2018;Zhu et al, 2020;Koto et al, 2021a) or unsupervised methods (Wu et al, 2020;Pandia et al, 2021) to analyze the syntactic structures (e.g., Hewitt and Manning (2019); Wu et al (2020)), relations (Papanikolaou et al, 2019), ontologies (Michael et al, 2020) and, to a more limited extend, discourse related behaviour (Zhu et al, 2020;Koto et al, 2021a;Pandia et al, 2021).…”

Discourse Probing of Pretrained Language Models

“…Upadhye et al (2020) analyzed how well two pretrained models capture referential biases of different classes of English verbs. Zhu et al (2020) applied the model of Feng and Hirst (2014) to parse IMDB documents (Maas et al, 2011) into discourse trees. Using this (potentially noisy) data, probing tasks were conducted by mapping attention layers into single vectors of document-level rhetorical features.…”