Discourse Probing of Pretrained Language Models

Koto, Fajri; Lau, Jey Han; Baldwin, Timothy

doi:10.18653/v1/2021.naacl-main.301

Cited by 23 publications

(31 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For instance, regarding the adaptability evaluation for Spanish models, Cañete et al (2020) recently proposed GLUES, a Spanish version of GLUE. In the case of representation evaluation, most of the work is in a cross-linguistic setting for word (S ¸ahin et al, 2020), sentence (Ravishankar et al, 2019) and discourse (Koto et al, 2021) evaluations. For this reason and following the motivation of works such as RuSen-tEval (Mikhailov et al, 2021), we provide SentEval and DiscoEval in Spanish, which consists of tasks originally created with texts in Spanish and aimed at evaluating models of that language.…”

Section: Language Model Evaluationsmentioning

confidence: 99%

Evaluation Benchmarks for Spanish Sentence Representations

Araujo¹,

Carvallo²,

Kundu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Due to the success of pre-trained language models, versions of languages other than English have been released in recent years. This fact implies the need for resources to evaluate these models. In the case of Spanish, there are few ways to systematically assess the models' quality. In this paper, we narrow the gap by building two evaluation benchmarks. Inspired by previous work (Conneau and Kiela, 2018;Chen et al., 2019), we introduce Spanish SentEval and Spanish DiscoEval, aiming to assess the capabilities of stand-alone and discourse-aware sentence representations, respectively. Our benchmarks include considerable pre-existing and newly constructed datasets that address different tasks from various domains. In addition, we evaluate and analyze the most recent pre-trained Spanish language models to exhibit their capabilities and limitations.As an example, we discover that for the case of discourse evaluation tasks, mBERT, a language model trained on multiple languages, usually provides a richer latent representation than models trained only with documents in Spanish. We hope our contribution will motivate a fairer, more comparable, and less cumbersome way to evaluate future Spanish language models.

show abstract

Section: Language Model Evaluationsmentioning

confidence: 99%

Evaluation Benchmarks for Spanish Sentence Representations

Araujo¹,

Carvallo²,

Kundu³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…At the base of our work are two of the most popular and frequently used PLMs: BERT (Devlin et al, 2018) and BART (Lewis et al, 2020). We choose these two popular approaches in our study due to their complementary nature (encoder-only vs. encoder-decoder) and based on previous work by Zhu et al (2020) and Koto et al (2021a), showing the effectiveness of BERT and BART models for discourse related tasks.…”

Section: Related Workmentioning

confidence: 99%

“…Besides their strong empirical results on most real-world problems, such as summarization (Zhang et al, 2020;Xiao et al, 2021a), question-answering (Joshi et al, 2020;Oguz et al, 2021) and sentiment analysis (Adhikari et al, 2019;, uncovering what kind of linguistic knowledge is captured by this new type of pre-trained language models (PLMs) has become a prominent question by itself. As part of this line of research, called BERTology (Rogers et al, 2020), researchers explore the amount of linguistic understanding encapsulated in PLMs, exposed through either external probing tasks (Raganato and Tiedemann, 2018;Zhu et al, 2020;Koto et al, 2021a) or unsupervised methods (Wu et al, 2020;Pandia et al, 2021) to analyze the syntactic structures (e.g., Hewitt and Manning (2019); Wu et al (2020)), relations (Papanikolaou et al, 2019), ontologies (Michael et al, 2020) and, to a more limited extend, discourse related behaviour (Zhu et al, 2020;Koto et al, 2021a;Pandia et al, 2021).…”

Section: Introductionmentioning

confidence: 99%

Towards Understanding Large-Scale Discourse Structures in Pre-Trained and Fine-Tuned Language Models

Huber¹,

Carenini²

2022

Preprint

View full text Add to dashboard Cite

With a growing number of BERTology work analyzing different components of pre-trained language models, we extend this line of research through an in-depth analysis of discourse information in pre-trained and finetuned language models. We move beyond prior work along three dimensions: First, we describe a novel approach to infer discourse structures from arbitrarily long documents. Second, we propose a new type of analysis to explore where and how accurately intrinsic discourse is captured in the BERT and BART models. Finally, we assess how similar the generated structures are to a variety of baselines as well as their distribution within and between models.

show abstract

“…Following this line, many subsequent papers queried the amount of knowledge from various parts of neural models. These included syntaxrelated (Lakretz et al, 2019;Hewitt and Manning, 2019), semantic-related (Tenney et al, 2019), and discourse-related information Koto et al, 2021). Towards developing reliable probing methods, several papers proposed control mechanisms (Pimentel et al, 2020;.…”

Section: Related Workmentioning

confidence: 99%

Quantifying the Task-Specific Information in Text-Based Classifications

Zhu,

Balagopalan,

Ghassemi

et al. 2021

Preprint

View full text Add to dashboard Cite

Recently, neural natural language models have attained state-of-the-art performance on a wide variety of tasks, but the high performance can result from superficial, surfacelevel cues (Bender and Koller, 2020;Niven and Kao, 2020). These surface cues, as the "shortcuts" inherent in the datasets, do not contribute to the task-specific information (TSI) of the classification tasks. While it is essential to look at the model performance, it is also important to understand the datasets. In this paper, we consider this question: Apart from the information introduced by the shortcut features, how much task-specific information is required to classify a dataset? We formulate this quantity in an information-theoretic framework. While this quantity is hard to compute, we approximate it with a fast and stable method. TSI quantifies the amount of linguistic knowledge modulo a set of predefined shortcuts -that contributes to classifying a sample from each dataset. This framework allows us to compare across datasets, saying that, apart from a set of "shortcut features", classifying each sample in the Multi-NLI task involves around 0.4 nats more TSI than in the Quora Question Pair.

show abstract

Discourse Probing of Pretrained Language Models

Cited by 23 publications

References 23 publications

Evaluation Benchmarks for Spanish Sentence Representations

Evaluation Benchmarks for Spanish Sentence Representations

Towards Understanding Large-Scale Discourse Structures in Pre-Trained and Fine-Tuned Language Models

Quantifying the Task-Specific Information in Text-Based Classifications

Contact Info

Product

Resources

About