Towards Understanding Large-Scale Discourse Structures in Pre-Trained and Fine-Tuned Language Models

Abstract

With a growing number of BERTology works analyzing different components of pre-trained language models, we extend this line of research through an in-depth analysis of discourse information in pre-trained and finetuned language models. We move beyond prior work along three dimensions: First, we describe a novel approach to infer discourse structures from arbitrarily long documents. Second, we propose a new type of analysis to explore where and how accurately intrinsic discourse is captured in the BERT and BART models. Finally, we assess how similar the generated structures are to a variety of baselines as well as their distributions within and between models.

Discourse Structure Extraction from Pre-Trained and Fine-Tuned Language Models in Dialogues

“…We explore both vanilla and fine-tuned PLMs, as they were both shown to contain discourse information for monologues (Huber and Carenini, 2022).…”

“…3.4 How to find the best heads? Xiao et al (2021) and Huber and Carenini (2022) showed that discourse information is not evenly distributed between heads and layers. However, they do not provide a strategy to select the head(s) containing most discourse information.…”

“…As a result, several transfer learning approaches have been proposed, mainly focused on monologues. Previous work demonstrate that discourse information can be extracted from auxiliary tasks like sentiment analysis and summarization (Xiao et al, 2021), or represented in language models (Koto et al, 2021) and further enhanced by fine-tuning tasks (Huber and Carenini, 2022). Inspired by the latter approaches, we are pioneering in addressing this issue for dialogues and introducing effective semi-supervised and unsupervised strategies to uncover discourse information in large pre-trained language models (PLMs).…”

An Integrated Approach for Political Bias Prediction and Explanation Based on Discursive Structure

“…We extend the use of such methods to take into account structural elements. Although some studies have recently investigated how structural / discourse information is encoded in pretrained languages models (Wu et al, 2020;Huber and Carenini, 2022), to the best of our knowledge, we are the first to explore textual explainable methods not relying only on surface form information. This is crucial for long texts, as methods such as LIME (Ribeiro et al, 2016) that rely on sampling word perturbations can become expensive for high token counts.…”