Temporal Effects on Pre-trained Models for Language Processing Tasks

Abstract

Keeping the performance of language technologies optimal as time passes is of great practical interest. Here we survey prior work concerned with the effect of time on system performance, establishing more nuanced terminology for discussing the topic and proper experimental design to support solid conclusions about the observed phenomena. We present a set of experiments with systems powered by large neural pretrained representations for English to demonstrate that temporal model deterioration is not as big a concern, with some models in fact improving when tested on data drawn from a later time period. It is however the case that temporal domain adaptation is beneficial, with better performance for a given time period possible when the system is trained on temporally more recent data. Our experiments reveal that the distinctions between temporal model deterioration and temporal domain adaptation becomes salient for systems built upon pretrained representations. Finally we examine the efficacy of two approaches for temporal domain adaptation without human annotations on new data, with self-labeling proving to be superior to continual pre-training. Notably, for named entity recognition, selflabeling leads to better temporal adaptation than human annotation.

TimeLMs: Diachronic Language Models from Twitter

“…A well-known case is the biomedical domain, e.g., BioBERT (Lee et al, 2020), SciBERT (Beltagy et al, 2019) or Pub-MedBERT (Gu et al, 2021). A similar approach to ours is probably the analysis of Agarwal and Nenkova (2021). This work showed how training language models in recent data can be beneficial, an improvement that was found to be marginal in Luu et al (2021) in different settings.…”

Temporal Attention for Language Models

“…Several recent studies have explored and evaluated the generalization ability of language models to time (Röttger and Pierrehumbert, 2021;Lazaridou et al, 2021;Agarwal and Nenkova, 2021). To better handle continuously evolving web content, Hombaiah et al ( 2021) performed incremental training.…”

“…The "static" nature of existing LMs makes them unaware of time, and in particular unware of language changes that occur over time. This prevents such models from adapting to time and generalizing temporally (Röttger and Pierrehumbert, 2021;Lazaridou et al, 2021;Hombaiah et al, 2021;Dhingra et al, 2021;Agarwal and Nenkova, 2021), abilities that were shown to be important for many tasks in NLP and Information Retrieval (Kanhabua and Anand, 2016;Rosin et al, 2017;Huang and Paul, 2019;Röttger and Pierrehumbert, 2021;Savov et al, 2021). Recently, to create time-aware models, the NLP community has started to use time as a feature in training and fine-tuning language models (Dhingra et al, 2021;Rosin et al, 2021).…”

Temporal Attention for Language Models

“…Several recent studies have explored and evaluated the generalization ability of language models to time (Röttger and Pierrehumbert, 2021;Lazaridou et al, 2021;Agarwal and Nenkova, 2021;Hofmann et al, 2021;Loureiro et al, 2022). To better handle continuously evolving web content, Hombaiah et al ( 2021) performed incremental training.…”

“…The "static" nature of existing LMs makes them unaware of time, and in particular unware of language changes that occur over time. This prevents such models from adapting to time and generalizing temporally (Röttger and Pierrehumbert, 2021;Lazaridou et al, 2021;Hombaiah et al, 2021;Dhingra et al, 2022;Agarwal and Nenkova, 2021;Loureiro et al, 2022), abilities that were shown to be important for many tasks in NLP and Information Retrieval (Kanhabua and Anand, 2016;Rosin et al, 2017;Huang and Paul, 2019;Röttger and Pierrehumbert, 2021;Savov et al, 2021). Recently, to create time-aware models, the NLP community has started to use time as a feature in training and fine-tuning language models (Dhingra et al, 2022;Rosin et al, 2022).…”