TimeLMs: Diachronic Language Models from Twitter

Loureiro, Daniel; Barbieri, Francesco; Neves, Leonardo; Espinosa-Anke, Luis; Camacho-Collados, José

doi:10.48550/arxiv.2202.03829

Cited by 26 publications

(32 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Broadly, most of the observed semantic shift can be described as changes in the popularity of different word senses [46]. Although this suggests that contextual language models [32] would be well-suited for mitigating the effect of semantic shift in longitudinal analyses, emerging research suggests this is not necessarily true in the absence of additional tuning [33,61].…”

Section: Naïvementioning

confidence: 99%

“…A lack of analyses of temporal robustness of these models belies the seriousness of the problem: language shifts over time -especially on social media [15,61] -and statistical classifiers degrade in the presence of distributional changes [28,50]. Three types of distributional change are of particular concern for classifiers applied over time: 1) new terminology is used to convey existing concepts; 2) existing terminology is used to convey new concepts; and 3) semantic relationships remain fixed, but the overall language distribution changes.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

The Problem of Semantic Shift in Longitudinal Monitoring of Social Media

Harrigian

Dredze

2022

14th ACM Web Science Conference 2022

View full text Add to dashboard Cite

Social media allows researchers to track societal and cultural changes over time based on language analysis tools. Many of these tools rely on statistical algorithms which need to be tuned to specific types of language. Recent studies have shown the absence of appropriate tuning, specifically in the presence of semantic shift, can hinder robustness of the underlying methods. However, little is known about the practical effect this sensitivity may have on downstream longitudinal analyses. We explore this gap in the literature through a timely case study: understanding shifts in depression during the course of the COVID-19 pandemic. We find that inclusion of only a small number of semantically-unstable features can promote significant changes in longitudinal estimates of our target outcome. At the same time, we demonstrate that a recently-introduced method for measuring semantic shift may be used to proactively identify failure points of language-based models and, in turn, improve predictive generalization.

show abstract

Section: Naïvementioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

The Problem of Semantic Shift in Longitudinal Monitoring of Social Media

Harrigian

Dredze

2022

14th ACM Web Science Conference 2022

View full text Add to dashboard Cite

show abstract

“…Several recent studies have explored and evaluated the generalization ability of language models to time (Röttger and Pierrehumbert, 2021;Lazaridou et al, 2021;Agarwal and Nenkova, 2021;Hofmann et al, 2021;Loureiro et al, 2022). To better handle continuously evolving web content, Hombaiah et al ( 2021) performed incremental training.…”

Section: Temporal Language Modelsmentioning

confidence: 99%

“…The "static" nature of existing LMs makes them unaware of time, and in particular unware of language changes that occur over time. This prevents such models from adapting to time and generalizing temporally (Röttger and Pierrehumbert, 2021;Lazaridou et al, 2021;Hombaiah et al, 2021;Dhingra et al, 2022;Agarwal and Nenkova, 2021;Loureiro et al, 2022), abilities that were shown to be important for many tasks in NLP and Information Retrieval (Kanhabua and Anand, 2016;Rosin et al, 2017;Huang and Paul, 2019;Röttger and Pierrehumbert, 2021;Savov et al, 2021). Recently, to create time-aware models, the NLP community has started to use time as a feature in training and fine-tuning language models (Dhingra et al, 2022;Rosin et al, 2022).…”

Section: Introductionmentioning

confidence: 99%

Temporal Attention for Language Models

Rosin¹,

Radinsky²

2022

Findings of the Association for Computational Linguistics: NAACL 2022

View full text Add to dashboard Cite

Pretrained language models based on the transformer architecture have shown great success in NLP. Textual training data often comes from the web and is thus tagged with time-specific information, but most language models ignore this information. They are trained on the textual data alone, limiting their ability to generalize temporally. In this work, we extend the key component of the transformer architecture, i.e., the self-attention mechanism, and propose temporal attention-a time-aware selfattention mechanism. Temporal attention can be applied to any transformer model and requires the input texts to be accompanied with their relevant time points. It allows the transformer to capture this temporal information and create time-specific contextualized word representations. We leverage these representations for the task of semantic change detection; we apply our proposed mechanism to BERT and experiment on three datasets in different languages (English, German, and Latin) that also vary in time, size, and genre. Our proposed model achieves state-of-the-art results on all the datasets.

show abstract

“…While these works focus on understanding bias in film directly, we take a slightly differently framing, examining how the bias in a film dataset can impact the biases of a language model. Loureiro et al (2022) examine concept drift and generalization on language models trained on Twitter data over time. Our work on longitudinal effects of film data is distinct in timescale (reflecting the much slower release rate of films relative to tweets) and in motivation; (Loureiro et al, 2022) consider the effects of the data's time period on model performance, while we examine the effects of the time period on model biases.…”

Section: Related Workmentioning

confidence: 99%

Evaluating Gender Bias Transfer from Film Data

Amanda¹,

Oh²,

Natu³

et al. 2022

Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP)

View full text Add to dashboard Cite

Films are a rich source of data for natural language processing. OpenSubtitles (Lison and Tiedemann, 2016) is a popular movie script dataset, used for training models for tasks such as machine translation and dialogue generation. However, movies often contain biases that reflect society at the time, and these biases may be introduced during pre-training and influence downstream models. We perform sentiment analysis on template infilling (Kurita et al., 2019) and the Sentence Embedding Association Test (May et al., 2019) to measure how BERT-based language models change after continued pre-training on OpenSubtitles. We consider gender bias as a primary motivating case for this analysis, while also measuring other social biases such as disability. We show that sentiment analysis on template infilling is not an effective measure of bias due to the rarity of disability and gender identifying tokens in the movie dialogue. We extend our analysis to a longitudinal study of bias in film dialogue over the last 110 years and find that continued pretraining on OpenSubtitles encodes additional bias into BERT. We show that BERT learns associations that reflect the biases and representation of each film era, suggesting that additional care must be taken when using historical data.

show abstract

TimeLMs: Diachronic Language Models from Twitter

Cited by 26 publications

References 26 publications

The Problem of Semantic Shift in Longitudinal Monitoring of Social Media

The Problem of Semantic Shift in Longitudinal Monitoring of Social Media

Temporal Attention for Language Models

Evaluating Gender Bias Transfer from Film Data

Contact Info

Product

Resources

About